mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
172e4d3a64
Part of #609. ## What was wrong On a rotated Android device `screenshot` returned a portrait-framed image with the content lying sideways. `describe` frames and gesture coordinates were **already upright** — uiautomator measures against the rotated display — so the capture was the one surface out of step. An agent reading that image saw geometry that disagreed with every coordinate it was given. Measured on a landscape Pixel_9 (API 36, `mCurrentOrientation=1`): | | space | |---|---| | `describe` | upright — `"Search Settings"` centre `(0.530, 0.234)` | | taps | upright — tapping describe's `"Display & touch"` centre opens it | | `screenshot` (default) | **portrait-native, sideways** | ## The fix simulator-server already applies `rotation` correctly; nothing was telling it which one. The rotation is now queried from the device per capture, so `screenshot`, `screenshot-diff`'s live captures and the auto-attached "Screen after action" screenshot all land in `describe`/tap space. **Nothing is tracked.** A cached orientation could only be staler than the ~40 ms probe, and serving a stale one is exactly the failure being fixed. When the rotation cannot be read the request is left byte-identical to before rather than guessing — a wrong orientation is a confidently wrong image, which is worse than a sideways one. **iOS is deliberately untouched.** There the whole surface is consistently portrait-native, so rotating only the capture would *break* an agreement rather than restore one. The iOS half of #609 needs a different change and is not in this PR. ### Also fixed: the legacy uiautomator describe path Same root cause, worse symptom. `wm size` reports the **unrotated** size — on the landscape Pixel_9 it still answered `Physical size: 1080x2424`. Dividing rotated bounds by that does not merely squash frames: `isVisibleRect` prunes nodes whose left edge is past the screen width, so **the right-hand half of a landscape screen vanished from the tree**. The dump carries `<hierarchy rotation="N">`, so the divisor is oriented from bytes already in hand — no extra round-trip on the path that runs precisely when the helper could not be reached. ## On the rotation-name table Pinned by measurement, never by what the names mean — the reporter of #609 found the same names are 180° inverted on iOS. `adb exec-out screencap` is rotation-aware, so it is ground truth; each candidate was scored as mean absolute difference against it both as-is and rotated 180°, so an inverted mapping could not pass: ``` rotation 1 LandscapeLeft 2.19 (vs 6.54 for its 180° twin) rotation 3 LandscapeRight 2.19 (vs 6.34) rotation 2 PortraitUpsideDown 2.67 (vs 13.13) rotation 0 Portrait identity ``` This table is the **inverse** of simulator-server's own `90° → LandscapeRight` convention, because it compensates for a rotation that layer applies while decoding the scrcpy stream. That makes it a property of a particular simulator-server build rather than of Android — which is why a table test alone is not enough, and the delivered PNG's aspect is checked against the rotation requested, falling back to an unrotated capture if they disagree. ## Verified live Branch tool-server against a real emulator: - landscape, default capture → **2424×1080 upright**, MAD 2.19 vs ground truth (6.54 vs its 180° twin) - cropping `describe`'s frame for the toolbar out of that default capture shows exactly the "Display & touch" toolbar — the two spaces agree - rotation 0 → 1080×2424, MAD 2.82, unchanged - rotated **iPad** → still 2064×2752 sideways, i.e. iOS provably untouched Plus 27 new unit tests, including one that asserts the legacy path *drops* the right-hand node before the fix and keeps it after. Full suite 3125 passing; lint, prettier, `typecheck:tests` and the tool-description gate all clean. ## Not in scope - The iOS half of #609. - `rotate` on Android silently no-ops when auto-rotate is enabled (it returned `{orientation:"LandscapeLeft"}` while `dumpsys` still read `cur=1080x2424`). Filing separately. - `screen-recording` still records sideways — same root cause, different pipeline. Worth noting a session's screenshots are now upright while its video is not. - Existing visual baselines captured on a rotated Android device will stop matching, since the baseline key embeds capture dimensions and those now swap. This fails loudly as a missing baseline rather than silently comparing the wrong thing — and those baselines were sideways captures of an upright UI. --- > **#673 and #658 are both stacked on this branch** — each conflicts with this one (`screenshot/index.ts` and `screenshot-diff/index.ts` respectively) but not with each other, so both were rebased onto this branch as siblings rather than chained. Merge this first; GitHub then retargets both to `main`. --------- Co-authored-by: Filip131311 <f.kaminski2000@gmail.com>
264 lines
11 KiB
TypeScript
264 lines
11 KiB
TypeScript
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
|
import { promises as fs } from "node:fs";
|
|
import * as os from "node:os";
|
|
import * as path from "node:path";
|
|
|
|
/**
|
|
* Issue #609: on a rotated Android device the screenshot came back
|
|
* portrait-framed with the content lying sideways, while `describe` frames and
|
|
* gesture coordinates were already upright. The capture was the one surface out
|
|
* of step.
|
|
*
|
|
* The rotation is queried per capture rather than tracked, so these tests are
|
|
* about reading it honestly and — crucially — about refusing to guess when it
|
|
* cannot be read. A wrong orientation yields a confidently wrong image, which is
|
|
* worse than the sideways one being fixed.
|
|
*/
|
|
|
|
const adbShell = vi.hoisted(() => vi.fn(async (_serial: string, _cmd: string) => ""));
|
|
vi.mock("../src/utils/adb", () => ({ adbShell }));
|
|
// The adb-server socket path is tried before the spawned client. Default it to
|
|
// "server not reachable" so the existing probe tests keep exercising the client.
|
|
const adbServerShell = vi.hoisted(() =>
|
|
vi.fn(async (_serial: string, _cmd: string): Promise<string> => {
|
|
throw new Error("ECONNREFUSED");
|
|
})
|
|
);
|
|
vi.mock("../src/utils/adb-server", () => ({ adbServerShell }));
|
|
|
|
import {
|
|
SURFACE_ROTATION_TO_NAME,
|
|
captureLooksUpright,
|
|
captureRotationForSurface,
|
|
readAndroidSurfaceRotation,
|
|
readPngSize,
|
|
} from "../src/utils/device-orientation";
|
|
|
|
/** Verbatim shape of the two probes, captured from a Pixel_9 (API 36). */
|
|
const DUMPSYS_DISPLAY = (r: number) => ` mCurrentOrientation=${r}\n`;
|
|
const DUMPSYS_WINDOW = (r: number) =>
|
|
`Display: mDisplayId=0\n init=1080x2424 420dpi\n mRotation=${r}\n mCurrentRotation=ROTATION_90\n`;
|
|
|
|
beforeEach(() => {
|
|
vi.clearAllMocks();
|
|
adbShell.mockResolvedValue("");
|
|
adbServerShell.mockRejectedValue(new Error("ECONNREFUSED"));
|
|
});
|
|
|
|
describe("the rotation is read from the cheapest source that answers", () => {
|
|
// Measured on a Pixel_9 emulator: helper peek ~1 ms, adb server socket ~8 ms,
|
|
// spawned adb client ~19 ms. All three report the same platform value, so
|
|
// the order only changes latency — these tests pin that order and that each
|
|
// tier's failure hands over to the next rather than becoming a guess.
|
|
it("prefers a peek that answers, and then touches no adb at all", async () => {
|
|
const peek = vi.fn(async () => 3);
|
|
expect(await readAndroidSurfaceRotation("emulator-5554", peek)).toBe(3);
|
|
expect(adbServerShell).not.toHaveBeenCalled();
|
|
expect(adbShell).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("falls through when the peek has no answer", async () => {
|
|
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554", async () => null)).toBe(1);
|
|
expect(adbServerShell).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("falls through when the peek throws or answers nonsense", async () => {
|
|
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(2));
|
|
expect(
|
|
await readAndroidSurfaceRotation("emulator-5554", async () => {
|
|
throw new Error("helper gone");
|
|
})
|
|
).toBe(2);
|
|
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(2));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554", async () => 7)).toBe(2);
|
|
});
|
|
|
|
it("uses the adb server socket before spawning a client", async () => {
|
|
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(1);
|
|
expect(adbServerShell).toHaveBeenCalledWith(
|
|
"emulator-5554",
|
|
expect.stringContaining("mCurrentOrientation"),
|
|
expect.anything()
|
|
);
|
|
expect(adbShell).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("tries the second probe over the socket before giving up on it", async () => {
|
|
adbServerShell.mockResolvedValueOnce("").mockResolvedValueOnce(DUMPSYS_WINDOW(3));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(3);
|
|
expect(adbServerShell).toHaveBeenCalledTimes(2);
|
|
expect(adbShell).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("spawns the client when the server socket cannot be used", async () => {
|
|
// e.g. the adb server is not running: the client will start it.
|
|
adbShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(1);
|
|
expect(adbServerShell).toHaveBeenCalledTimes(2);
|
|
expect(adbShell).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("still refuses to guess when every source is silent", async () => {
|
|
expect(await readAndroidSurfaceRotation("emulator-5554", async () => undefined)).toBeNull();
|
|
});
|
|
});
|
|
|
|
describe("the surface-rotation → rotation-name table", () => {
|
|
// Pinned by measurement against `adb exec-out screencap`, which IS
|
|
// rotation-aware: each candidate name was scored as mean absolute difference
|
|
// against the screencap reference and against that reference rotated 180°, so
|
|
// an inverted mapping could not pass. The correct name won by ~3x in every row
|
|
// (e.g. rotation 1: LandscapeLeft 2.19 vs LandscapeRight 6.54).
|
|
//
|
|
// This table is the INVERSE of simulator-server's own 90°→LandscapeRight
|
|
// convention because we are compensating for a rotation it applies while
|
|
// decoding. That is exactly why this test cannot be the only guard — see
|
|
// `captureLooksUpright`, which checks the real capture instead.
|
|
it("maps each rotation to the name that yields an upright capture", () => {
|
|
expect(SURFACE_ROTATION_TO_NAME).toEqual({
|
|
0: "Portrait",
|
|
1: "LandscapeLeft",
|
|
2: "PortraitUpsideDown",
|
|
3: "LandscapeRight",
|
|
});
|
|
});
|
|
|
|
it("sends no rotation at all for an unrotated device", () => {
|
|
// Not "Portrait": keeping it undefined leaves the request body byte-identical
|
|
// to what it was before any of this existed, so the common case is provably
|
|
// unchanged.
|
|
expect(captureRotationForSurface(0)).toBeUndefined();
|
|
});
|
|
|
|
it("sends no rotation when the rotation could not be read", () => {
|
|
expect(captureRotationForSurface(null)).toBeUndefined();
|
|
});
|
|
|
|
it("requests a rotation for each side the device can be on", () => {
|
|
expect(captureRotationForSurface(1)).toBe("LandscapeLeft");
|
|
expect(captureRotationForSurface(2)).toBe("PortraitUpsideDown");
|
|
expect(captureRotationForSurface(3)).toBe("LandscapeRight");
|
|
});
|
|
});
|
|
|
|
describe("reading the rotation off the device", () => {
|
|
it("reads the primary probe", async () => {
|
|
adbShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(1);
|
|
expect(adbShell).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("reads every rotation the platform can report", async () => {
|
|
for (const r of [0, 1, 2, 3]) {
|
|
adbShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(r));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(r);
|
|
}
|
|
});
|
|
|
|
it("falls back to the second probe when the first says nothing", async () => {
|
|
// `dumpsys` output is not a stable API, so a single regex is a single point
|
|
// of failure across vendors and versions.
|
|
adbShell.mockResolvedValueOnce("").mockResolvedValueOnce(DUMPSYS_WINDOW(3));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(3);
|
|
expect(adbShell).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
it("keeps a grep miss from throwing", async () => {
|
|
// The probe appends `|| true` precisely so that grep exiting 1 — the normal
|
|
// outcome when the line is absent — is not an error.
|
|
expect(adbShell.mock.calls).toHaveLength(0);
|
|
await readAndroidSurfaceRotation("emulator-5554");
|
|
for (const [, cmd] of adbShell.mock.calls) expect(cmd).toContain("|| true");
|
|
});
|
|
|
|
it("returns null rather than guessing when both probes fail", async () => {
|
|
adbShell.mockRejectedValue(new Error("device offline"));
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBeNull();
|
|
});
|
|
|
|
it("returns null on unparseable output", async () => {
|
|
adbShell.mockResolvedValue("mCurrentOrientation=banana");
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBeNull();
|
|
});
|
|
|
|
it("ignores an out-of-range rotation", async () => {
|
|
// Never coerce something unrecognised into a rotation — a wrong one is worse
|
|
// than none.
|
|
adbShell.mockResolvedValue("mCurrentOrientation=7");
|
|
expect(await readAndroidSurfaceRotation("emulator-5554")).toBeNull();
|
|
});
|
|
});
|
|
|
|
describe("the aspect guard on the delivered capture", () => {
|
|
// The mapping compensates for what simulator-server does at decode time. If
|
|
// that changes, the compensation silently becomes a 180° error or a no-op and
|
|
// no test of our own constant could see it. This checks the actual image.
|
|
it("accepts a landscape image for a landscape rotation", () => {
|
|
expect(captureLooksUpright("LandscapeLeft", { width: 2424, height: 1080 })).toBe(true);
|
|
expect(captureLooksUpright("LandscapeRight", { width: 2424, height: 1080 })).toBe(true);
|
|
});
|
|
|
|
it("rejects a portrait image delivered for a landscape rotation", () => {
|
|
expect(captureLooksUpright("LandscapeLeft", { width: 1080, height: 2424 })).toBe(false);
|
|
});
|
|
|
|
it("accepts a portrait image for an upside-down portrait rotation", () => {
|
|
expect(captureLooksUpright("PortraitUpsideDown", { width: 1080, height: 2424 })).toBe(true);
|
|
});
|
|
|
|
it("does not treat an unreadable image as evidence of a problem", () => {
|
|
expect(captureLooksUpright("LandscapeLeft", null)).toBe(true);
|
|
});
|
|
|
|
it("passes a square image, which carries no aspect information", () => {
|
|
expect(captureLooksUpright("LandscapeLeft", { width: 512, height: 512 })).toBe(true);
|
|
});
|
|
});
|
|
|
|
describe("readPngSize", () => {
|
|
let dir: string;
|
|
|
|
beforeEach(async () => {
|
|
dir = await fs.mkdtemp(path.join(os.tmpdir(), "argent-png-size-"));
|
|
});
|
|
afterEach(async () => {
|
|
await fs.rm(dir, { recursive: true, force: true });
|
|
});
|
|
|
|
/** A minimal valid PNG header — signature, then IHDR length/type/w/h. */
|
|
function pngHeader(width: number, height: number): Buffer {
|
|
const buf = Buffer.alloc(24);
|
|
buf.writeUInt32BE(0x89504e47, 0);
|
|
buf.writeUInt32BE(0x0d0a1a0a, 4);
|
|
buf.writeUInt32BE(13, 8);
|
|
buf.write("IHDR", 12, "ascii");
|
|
buf.writeUInt32BE(width, 16);
|
|
buf.writeUInt32BE(height, 20);
|
|
return buf;
|
|
}
|
|
|
|
it("reads the dimensions out of the IHDR chunk", async () => {
|
|
const file = path.join(dir, "a.png");
|
|
await fs.writeFile(file, pngHeader(2424, 1080));
|
|
expect(await readPngSize(file)).toEqual({ width: 2424, height: 1080 });
|
|
});
|
|
|
|
it("returns null for a file that is not a PNG", async () => {
|
|
const file = path.join(dir, "b.png");
|
|
await fs.writeFile(file, Buffer.alloc(24));
|
|
expect(await readPngSize(file)).toBeNull();
|
|
});
|
|
|
|
it("returns null for a truncated file rather than throwing", async () => {
|
|
const file = path.join(dir, "c.png");
|
|
await fs.writeFile(file, Buffer.alloc(8));
|
|
expect(await readPngSize(file)).toBeNull();
|
|
});
|
|
|
|
it("returns null for a missing file rather than throwing", async () => {
|
|
expect(await readPngSize(path.join(dir, "nope.png"))).toBeNull();
|
|
});
|
|
});
|