mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
172e4d3a64
Part of #609. ## What was wrong On a rotated Android device `screenshot` returned a portrait-framed image with the content lying sideways. `describe` frames and gesture coordinates were **already upright** — uiautomator measures against the rotated display — so the capture was the one surface out of step. An agent reading that image saw geometry that disagreed with every coordinate it was given. Measured on a landscape Pixel_9 (API 36, `mCurrentOrientation=1`): | | space | |---|---| | `describe` | upright — `"Search Settings"` centre `(0.530, 0.234)` | | taps | upright — tapping describe's `"Display & touch"` centre opens it | | `screenshot` (default) | **portrait-native, sideways** | ## The fix simulator-server already applies `rotation` correctly; nothing was telling it which one. The rotation is now queried from the device per capture, so `screenshot`, `screenshot-diff`'s live captures and the auto-attached "Screen after action" screenshot all land in `describe`/tap space. **Nothing is tracked.** A cached orientation could only be staler than the ~40 ms probe, and serving a stale one is exactly the failure being fixed. When the rotation cannot be read the request is left byte-identical to before rather than guessing — a wrong orientation is a confidently wrong image, which is worse than a sideways one. **iOS is deliberately untouched.** There the whole surface is consistently portrait-native, so rotating only the capture would *break* an agreement rather than restore one. The iOS half of #609 needs a different change and is not in this PR. ### Also fixed: the legacy uiautomator describe path Same root cause, worse symptom. `wm size` reports the **unrotated** size — on the landscape Pixel_9 it still answered `Physical size: 1080x2424`. Dividing rotated bounds by that does not merely squash frames: `isVisibleRect` prunes nodes whose left edge is past the screen width, so **the right-hand half of a landscape screen vanished from the tree**. The dump carries `<hierarchy rotation="N">`, so the divisor is oriented from bytes already in hand — no extra round-trip on the path that runs precisely when the helper could not be reached. ## On the rotation-name table Pinned by measurement, never by what the names mean — the reporter of #609 found the same names are 180° inverted on iOS. `adb exec-out screencap` is rotation-aware, so it is ground truth; each candidate was scored as mean absolute difference against it both as-is and rotated 180°, so an inverted mapping could not pass: ``` rotation 1 LandscapeLeft 2.19 (vs 6.54 for its 180° twin) rotation 3 LandscapeRight 2.19 (vs 6.34) rotation 2 PortraitUpsideDown 2.67 (vs 13.13) rotation 0 Portrait identity ``` This table is the **inverse** of simulator-server's own `90° → LandscapeRight` convention, because it compensates for a rotation that layer applies while decoding the scrcpy stream. That makes it a property of a particular simulator-server build rather than of Android — which is why a table test alone is not enough, and the delivered PNG's aspect is checked against the rotation requested, falling back to an unrotated capture if they disagree. ## Verified live Branch tool-server against a real emulator: - landscape, default capture → **2424×1080 upright**, MAD 2.19 vs ground truth (6.54 vs its 180° twin) - cropping `describe`'s frame for the toolbar out of that default capture shows exactly the "Display & touch" toolbar — the two spaces agree - rotation 0 → 1080×2424, MAD 2.82, unchanged - rotated **iPad** → still 2064×2752 sideways, i.e. iOS provably untouched Plus 27 new unit tests, including one that asserts the legacy path *drops* the right-hand node before the fix and keeps it after. Full suite 3125 passing; lint, prettier, `typecheck:tests` and the tool-description gate all clean. ## Not in scope - The iOS half of #609. - `rotate` on Android silently no-ops when auto-rotate is enabled (it returned `{orientation:"LandscapeLeft"}` while `dumpsys` still read `cur=1080x2424`). Filing separately. - `screen-recording` still records sideways — same root cause, different pipeline. Worth noting a session's screenshots are now upright while its video is not. - Existing visual baselines captured on a rotated Android device will stop matching, since the baseline key embeds capture dimensions and those now swap. This fails loudly as a missing baseline rather than silently comparing the wrong thing — and those baselines were sideways captures of an upright UI. --- > **#673 and #658 are both stacked on this branch** — each conflicts with this one (`screenshot/index.ts` and `screenshot-diff/index.ts` respectively) but not with each other, so both were rebased onto this branch as siblings rather than chained. Merge this first; GitHub then retargets both to `main`. --------- Co-authored-by: Filip131311 <f.kaminski2000@gmail.com>
143 lines
5.4 KiB
TypeScript
143 lines
5.4 KiB
TypeScript
import { describe, it, expect, vi, beforeEach } from "vitest";
|
|
import type { DeviceInfo } from "@argent/registry";
|
|
|
|
/**
|
|
* Issue #609: the capture is the surface that has to move, not the coordinates.
|
|
* On Android `describe` frames and gesture input are already upright, so the
|
|
* capture is made to follow the device rotation to match them. iOS is left
|
|
* alone: there the whole surface is consistently portrait-native, and rotating
|
|
* only the capture would break an agreement rather than restore one.
|
|
*/
|
|
|
|
const readAndroidSurfaceRotation = vi.hoisted(() =>
|
|
vi.fn(async (_serial: string): Promise<0 | 1 | 2 | 3 | null> => null)
|
|
);
|
|
const readPngSize = vi.hoisted(() =>
|
|
vi.fn(async (_p: string): Promise<{ width: number; height: number } | null> => null)
|
|
);
|
|
|
|
vi.mock("../src/utils/device-orientation", async (importOriginal) => {
|
|
// The mapping table and the aspect guard stay REAL so the assertions below
|
|
// exercise the actual constant rather than a restatement of it.
|
|
const actual = await importOriginal<typeof import("../src/utils/device-orientation")>();
|
|
return { ...actual, readAndroidSurfaceRotation, readPngSize };
|
|
});
|
|
|
|
import { captureScreenshotUpright } from "../src/utils/rotation-aware-capture";
|
|
|
|
const ANDROID: DeviceInfo = { id: "emulator-5554", platform: "android", kind: "emulator" };
|
|
const IOS: DeviceInfo = { id: "BC0026C7-AAE0-490E", platform: "ios", kind: "simulator" };
|
|
|
|
const api = {} as never;
|
|
|
|
/** Records the rotation each capture was asked for. */
|
|
function recorder(size: { width: number; height: number } = { width: 2424, height: 1080 }) {
|
|
const rotations: (string | undefined)[] = [];
|
|
const capture = vi.fn(async (_api: unknown, rotation?: string) => {
|
|
rotations.push(rotation);
|
|
return { url: "http://x/y.png", path: "/tmp/y.png" };
|
|
});
|
|
readPngSize.mockResolvedValue(size);
|
|
return { rotations, capture: capture as never };
|
|
}
|
|
|
|
beforeEach(() => {
|
|
vi.clearAllMocks();
|
|
readAndroidSurfaceRotation.mockResolvedValue(null);
|
|
readPngSize.mockResolvedValue(null);
|
|
});
|
|
|
|
describe("Android capture follows the device rotation", () => {
|
|
it("requests the matching rotation for a landscape device", async () => {
|
|
readAndroidSurfaceRotation.mockResolvedValue(1);
|
|
const { rotations, capture } = recorder();
|
|
|
|
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
|
|
|
|
expect(rotations).toEqual(["LandscapeLeft"]);
|
|
});
|
|
|
|
it("sends no rotation for an unrotated device", async () => {
|
|
// The overwhelmingly common case must produce the exact request it always
|
|
// did — not `rotation: "Portrait"`.
|
|
readAndroidSurfaceRotation.mockResolvedValue(0);
|
|
const { rotations, capture } = recorder({ width: 1080, height: 2424 });
|
|
|
|
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
|
|
|
|
expect(rotations).toEqual([undefined]);
|
|
});
|
|
|
|
it("sends no rotation when the device would not say", async () => {
|
|
readAndroidSurfaceRotation.mockResolvedValue(null);
|
|
const { rotations, capture } = recorder({ width: 1080, height: 2424 });
|
|
|
|
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
|
|
|
|
expect(rotations).toEqual([undefined]);
|
|
});
|
|
|
|
it("lets an explicit rotation win, and does not probe at all", async () => {
|
|
const { rotations, capture } = recorder();
|
|
|
|
await captureScreenshotUpright(
|
|
api,
|
|
ANDROID,
|
|
"PortraitUpsideDown",
|
|
undefined,
|
|
undefined,
|
|
capture
|
|
);
|
|
|
|
expect(rotations).toEqual(["PortraitUpsideDown"]);
|
|
expect(readAndroidSurfaceRotation).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("passes the scale through unchanged", async () => {
|
|
readAndroidSurfaceRotation.mockResolvedValue(1);
|
|
const { capture } = recorder();
|
|
|
|
await captureScreenshotUpright(api, ANDROID, undefined, undefined, 1.0, capture);
|
|
|
|
expect(capture).toHaveBeenCalledWith(api, "LandscapeLeft", undefined, 1.0);
|
|
});
|
|
});
|
|
|
|
describe("iOS is deliberately untouched", () => {
|
|
it("never probes and never adds a rotation", async () => {
|
|
const { rotations, capture } = recorder();
|
|
|
|
await captureScreenshotUpright(api, IOS, undefined, undefined, undefined, capture);
|
|
|
|
expect(rotations).toEqual([undefined]);
|
|
expect(readAndroidSurfaceRotation).not.toHaveBeenCalled();
|
|
});
|
|
});
|
|
|
|
describe("the aspect guard", () => {
|
|
it("falls back to an unrotated capture when the image comes back the wrong shape", async () => {
|
|
// Simulates simulator-server's rotation handling changing underneath us: we
|
|
// ask for landscape and get a portrait-shaped PNG. Shipping that would be a
|
|
// confidently wrong image, so the unrotated capture is preferred.
|
|
readAndroidSurfaceRotation.mockResolvedValue(1);
|
|
const { rotations, capture } = recorder({ width: 1080, height: 2424 });
|
|
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
|
|
|
|
expect(rotations).toEqual(["LandscapeLeft", undefined]);
|
|
expect(warn).toHaveBeenCalled();
|
|
warn.mockRestore();
|
|
});
|
|
|
|
it("does not second-guess a capture whose size cannot be read", async () => {
|
|
readAndroidSurfaceRotation.mockResolvedValue(1);
|
|
const { rotations, capture } = recorder();
|
|
readPngSize.mockResolvedValue(null);
|
|
|
|
await captureScreenshotUpright(api, ANDROID, undefined, undefined, undefined, capture);
|
|
|
|
expect(rotations).toEqual(["LandscapeLeft"]);
|
|
});
|
|
});
|