Files
software-mansion__argent/packages/tool-server/test/rotation-aware-capture.test.ts
filip131311 172e4d3a64 fix(android): capture the screen the way it is actually oriented (#693)
Part of #609.

## What was wrong

On a rotated Android device `screenshot` returned a portrait-framed
image with the content lying sideways. `describe` frames and gesture
coordinates were **already upright** — uiautomator measures against the
rotated display — so the capture was the one surface out of step. An
agent reading that image saw geometry that disagreed with every
coordinate it was given.

Measured on a landscape Pixel_9 (API 36, `mCurrentOrientation=1`):

| | space |
|---|---|
| `describe` | upright — `"Search Settings"` centre `(0.530, 0.234)` |
| taps | upright — tapping describe's `"Display & touch"` centre opens
it |
| `screenshot` (default) | **portrait-native, sideways** |

## The fix

simulator-server already applies `rotation` correctly; nothing was
telling it which one. The rotation is now queried from the device per
capture, so `screenshot`, `screenshot-diff`'s live captures and the
auto-attached "Screen after action" screenshot all land in
`describe`/tap space.

**Nothing is tracked.** A cached orientation could only be staler than
the ~40 ms probe, and serving a stale one is exactly the failure being
fixed. When the rotation cannot be read the request is left
byte-identical to before rather than guessing — a wrong orientation is a
confidently wrong image, which is worse than a sideways one.

**iOS is deliberately untouched.** There the whole surface is
consistently portrait-native, so rotating only the capture would *break*
an agreement rather than restore one. The iOS half of #609 needs a
different change and is not in this PR.

### Also fixed: the legacy uiautomator describe path

Same root cause, worse symptom. `wm size` reports the **unrotated** size
— on the landscape Pixel_9 it still answered `Physical size: 1080x2424`.
Dividing rotated bounds by that does not merely squash frames:
`isVisibleRect` prunes nodes whose left edge is past the screen width,
so **the right-hand half of a landscape screen vanished from the tree**.
The dump carries `<hierarchy rotation="N">`, so the divisor is oriented
from bytes already in hand — no extra round-trip on the path that runs
precisely when the helper could not be reached.

## On the rotation-name table

Pinned by measurement, never by what the names mean — the reporter of
#609 found the same names are 180° inverted on iOS. `adb exec-out
screencap` is rotation-aware, so it is ground truth; each candidate was
scored as mean absolute difference against it both as-is and rotated
180°, so an inverted mapping could not pass:

```
rotation 1  LandscapeLeft       2.19  (vs 6.54 for its 180° twin)
rotation 3  LandscapeRight      2.19  (vs 6.34)
rotation 2  PortraitUpsideDown  2.67  (vs 13.13)
rotation 0  Portrait            identity
```

This table is the **inverse** of simulator-server's own `90° →
LandscapeRight` convention, because it compensates for a rotation that
layer applies while decoding the scrcpy stream. That makes it a property
of a particular simulator-server build rather than of Android — which is
why a table test alone is not enough, and the delivered PNG's aspect is
checked against the rotation requested, falling back to an unrotated
capture if they disagree.

## Verified live

Branch tool-server against a real emulator:

- landscape, default capture → **2424×1080 upright**, MAD 2.19 vs ground
truth (6.54 vs its 180° twin)
- cropping `describe`'s frame for the toolbar out of that default
capture shows exactly the "Display & touch" toolbar — the two spaces
agree
- rotation 0 → 1080×2424, MAD 2.82, unchanged
- rotated **iPad** → still 2064×2752 sideways, i.e. iOS provably
untouched

Plus 27 new unit tests, including one that asserts the legacy path
*drops* the right-hand node before the fix and keeps it after. Full
suite 3125 passing; lint, prettier, `typecheck:tests` and the
tool-description gate all clean.

## Not in scope

- The iOS half of #609.
- `rotate` on Android silently no-ops when auto-rotate is enabled (it
returned `{orientation:"LandscapeLeft"}` while `dumpsys` still read
`cur=1080x2424`). Filing separately.
- `screen-recording` still records sideways — same root cause, different
pipeline. Worth noting a session's screenshots are now upright while its
video is not.
- Existing visual baselines captured on a rotated Android device will
stop matching, since the baseline key embeds capture dimensions and
those now swap. This fails loudly as a missing baseline rather than
silently comparing the wrong thing — and those baselines were sideways
captures of an upright UI.

---

> **#673 and #658 are both stacked on this branch** — each conflicts
with this one (`screenshot/index.ts` and `screenshot-diff/index.ts`
respectively) but not with each other, so both were rebased onto this
branch as siblings rather than chained. Merge this first; GitHub then
retargets both to `main`.

---------

Co-authored-by: Filip131311 <f.kaminski2000@gmail.com>
2026-08-21 17:34:59 +02:00

143 lines
5.4 KiB
TypeScript

import { describe, it, expect, vi, beforeEach } from "vitest";
import type { DeviceInfo } from "@argent/registry";
/**
* Issue #609: the capture is the surface that has to move, not the coordinates.
* On Android `describe` frames and gesture input are already upright, so the
* capture is made to follow the device rotation to match them. iOS is left
* alone: there the whole surface is consistently portrait-native, and rotating
* only the capture would break an agreement rather than restore one.
*/
const readAndroidSurfaceRotation = vi.hoisted(() =>
vi.fn(async (_serial: string): Promise<0 | 1 | 2 | 3 | null> => null)
);
const readPngSize = vi.hoisted(() =>
vi.fn(async (_p: string): Promise<{ width: number; height: number } | null> => null)
);
vi.mock("../src/utils/device-orientation", async (importOriginal) => {
// The mapping table and the aspect guard stay REAL so the assertions below
// exercise the actual constant rather than a restatement of it.
const actual = await importOriginal<typeof import("../src/utils/device-orientation")>();
return { ...actual, readAndroidSurfaceRotation, readPngSize };
});
import { captureScreenshotUpright } from "../src/utils/rotation-aware-capture";
const ANDROID: DeviceInfo = { id: "emulator-5554", platform: "android", kind: "emulator" };
const IOS: DeviceInfo = { id: "BC0026C7-AAE0-490E", platform: "ios", kind: "simulator" };
const api = {} as never;
/** Records the rotation each capture was asked for. */
function recorder(size: { width: number; height: number } = { width: 2424, height: 1080 }) {
const rotations: (string | undefined)[] = [];
const capture = vi.fn(async (_api: unknown, rotation?: string) => {
rotations.push(rotation);
return { url: "http://x/y.png", path: "/tmp/y.png" };
});
readPngSize.mockResolvedValue(size);
return { rotations, capture: capture as never };
}
beforeEach(() => {
vi.clearAllMocks();
readAndroidSurfaceRotation.mockResolvedValue(null);
readPngSize.mockResolvedValue(null);
});
describe("Android capture follows the device rotation", () => {
it("requests the matching rotation for a landscape device", async () => {
readAndroidSurfaceRotation.mockResolvedValue(1);
const { rotations, capture } = recorder();
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
expect(rotations).toEqual(["LandscapeLeft"]);
});
it("sends no rotation for an unrotated device", async () => {
// The overwhelmingly common case must produce the exact request it always
// did — not `rotation: "Portrait"`.
readAndroidSurfaceRotation.mockResolvedValue(0);
const { rotations, capture } = recorder({ width: 1080, height: 2424 });
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
expect(rotations).toEqual([undefined]);
});
it("sends no rotation when the device would not say", async () => {
readAndroidSurfaceRotation.mockResolvedValue(null);
const { rotations, capture } = recorder({ width: 1080, height: 2424 });
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
expect(rotations).toEqual([undefined]);
});
it("lets an explicit rotation win, and does not probe at all", async () => {
const { rotations, capture } = recorder();
await captureScreenshotUpright(
api,
ANDROID,
"PortraitUpsideDown",
undefined,
undefined,
capture
);
expect(rotations).toEqual(["PortraitUpsideDown"]);
expect(readAndroidSurfaceRotation).not.toHaveBeenCalled();
});
it("passes the scale through unchanged", async () => {
readAndroidSurfaceRotation.mockResolvedValue(1);
const { capture } = recorder();
await captureScreenshotUpright(api, ANDROID, undefined, undefined, 1.0, capture);
expect(capture).toHaveBeenCalledWith(api, "LandscapeLeft", undefined, 1.0);
});
});
describe("iOS is deliberately untouched", () => {
it("never probes and never adds a rotation", async () => {
const { rotations, capture } = recorder();
await captureScreenshotUpright(api, IOS, undefined, undefined, undefined, capture);
expect(rotations).toEqual([undefined]);
expect(readAndroidSurfaceRotation).not.toHaveBeenCalled();
});
});
describe("the aspect guard", () => {
it("falls back to an unrotated capture when the image comes back the wrong shape", async () => {
// Simulates simulator-server's rotation handling changing underneath us: we
// ask for landscape and get a portrait-shaped PNG. Shipping that would be a
// confidently wrong image, so the unrotated capture is preferred.
readAndroidSurfaceRotation.mockResolvedValue(1);
const { rotations, capture } = recorder({ width: 1080, height: 2424 });
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
expect(rotations).toEqual(["LandscapeLeft", undefined]);
expect(warn).toHaveBeenCalled();
warn.mockRestore();
});
it("does not second-guess a capture whose size cannot be read", async () => {
readAndroidSurfaceRotation.mockResolvedValue(1);
const { rotations, capture } = recorder();
readPngSize.mockResolvedValue(null);
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
expect(rotations).toEqual(["LandscapeLeft"]);
});
});