Files
Jakub Piasecki 8062da77a5 fix(flow): settle before gestures that resolve no selector (#779)
A selector target settles inside `waitForFrame`, but a target that
resolves nothing skipped the settle altogether: a coordinate
`tap`/`long-press` and a centre-anchored `pinch`/`rotate` dispatched
into whatever motion was in flight, so they could land mid-fling, which
is the exact race the settle exists to close. Every input directive now
settles first, however it is addressed.

The settle is best-effort here, unlike on the selector path: these
gestures read no frame out of the tree, so an outage must not fail them,
which would break the escape hatch coordinates exist to be (an element
the tree cannot see). This is the allowance `snapshot` already makes
before a capture that reads pixels rather than nodes.

Best-effort is not free, though, and the run pays it per gesture. A tree
the run can never read is not only the disconnect mid-run:
`fetchFlowTree` serves no tree at all on `ios-remote`, and an app that
cannot load the instrumentation fails every read off an in-memory list,
an Apple system app being the case in point, which flows drive by
coordinates for exactly that reason, so every step of such a flow is one
of these gestures. Measured against the real fetch on an `ios-remote`
device, a coordinate tap went from 0ms to 3022ms, once per gesture, for
a settle that was never going to arrive.

So a settle that spends its whole window without one successful read
records the outage on the run, against the device that proved it, and a
later gesture rethrows that instead of buying the same verdict again.
Not a shorter budget: any threshold low enough to spare a source that
serves no tree would also abandon the mid-navigation blip the settle's
retry exists for. But the record is a prediction, not a proof of
repetition. One window mints it and nothing re-tests it, and on iOS that
window can be a single stalled read, since the deadline is only tested
after a read returns and `ViewHierarchy.getFullHierarchy` sits on a 15s
RPC tier behind a serial 5s `Application.getState` probe. Being wrong
therefore costs a settle rather than a step, and it is not silent: a
gesture that spends the record warns its step report that it dispatched
unsettled, quoting the source's own reason, so a run whose prediction
was wrong says which of its greens mean the gesture was sent rather than
that it landed.

Every tree read a directive makes now goes through one helper so that a
read which comes back clears the record whichever directive asked for
it. `await`/`assert` and `idle` never settle, and a run that reads the
tree successfully must not keep skipping. A relaunch clears it too,
whether written as a `launch:` directive or as one of the raw `tool:`
steps that perform the same operation, since a relaunch is the repair
the commonest of these errors asks for by name. Two more reads route
through the same helper for uniformity but can never be the clearing
one: `rotate`'s aspect read skips itself while a record stands, and the
`type` focus wait runs only behind a `waitForFrame` whose own settle
already cleared it.

What it costs, in two directions. An outage that lasted a full window,
in a run that then never reads the tree again and never relaunches,
leaves later gestures unsettled even once it clears: a flow of nothing
but coordinate gestures giving up a best-effort settle it never had
before this change, against every one of those gestures otherwise paying
the window for a settle that is not coming. And on a screen that never
holds still, a spinner, a video, a ticking clock, nothing converges, so
every one of these gestures pays the whole window, measured at about
3.05s each. The record buys no relief there by design, because a window
that read the tree proves no outage, and the fingerprint cannot be
narrowed the way `scrollToVisible` narrows its own: a scroll knows the
motion it waits on is its own, inside the containers under its anchor,
while a gesture waits on motion of unknown origin that can move what
sits under the point without touching it. It is the same window every
selector directive already paid on such a screen; what changed is that
coordinate targets pay it too.

Two smaller pieces ride along. A centre-anchored `rotate` skips its
screen-aspect read on a recorded outage as well, degrading to the same
legacy orbit a failed read gives it, so the record actually takes the
window off that directive instead of handing back a 15s-tier RPC. And
the settle's tree read now takes the run's abort signal, so a cancelled
run reports its skip immediately rather than once the source finally
answers. It deliberately takes no time budget: one cut to what is left
of the window would fail the cold start #778 raised that tier to ride
out.
2026-08-17 11:15:47 +02:00

467 lines
17 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import type { Registry } from "@argent/registry";
import type { DescribeNode, DescribeTreeData } from "../../src/tools/describe/contract";
// Serve the flow tree directly: flows resolve selectors against the platform's
// full-hierarchy source and hard-fail rather than degrade to the AX tree, so
// these unit tests stub the tree fetch itself. `currentScreen` feeds the
// optional pixel dimensions the rotate directive's physical-circle math reads.
let currentTree: () => DescribeNode;
let currentScreen: () => { width: number; height: number } | undefined;
vi.mock("../../src/tools/flows/flow-tree", async (importOriginal) => ({
...(await importOriginal<typeof import("../../src/tools/flows/flow-tree")>()),
fetchFlowTree: vi.fn(async (): Promise<DescribeTreeData> => {
const screen = currentScreen();
return { tree: currentTree(), source: "native-devtools", ...(screen ? { screen } : {}) };
}),
}));
import { createRunFlowTool, type FlowRunResult } from "../../src/tools/flows/flow-run";
import { serializeFlow, parseFlow } from "../../src/tools/flows/flow-utils";
const DEVICE = "00000000-0000-0000-0000-0000000000ab"; // iOS UDID shape
const SCREEN_W = 1080;
const SCREEN_H = 2400;
let tmpDir: string;
function n(partial: Partial<DescribeNode> & { frame: DescribeNode["frame"] }): DescribeNode {
return { role: "AXOther", children: [], ...partial };
}
function screen(children: DescribeNode[]): DescribeNode {
return n({ role: "AXWindow", frame: { x: 0, y: 0, width: 1, height: 1 }, children });
}
/** Registry that records every gesture tool invocation's args. */
function mockRegistry(calls: Array<{ tool: string; args: Record<string, unknown> }>): Registry {
return {
invokeTool: vi.fn(async (id: string, args: Record<string, unknown>) => {
if (id === "list-devices") return { devices: [] };
calls.push({ tool: id, args });
return { ok: true };
}),
getTool: vi.fn(() => ({ inputSchema: { properties: { udid: {} } } })),
} as unknown as Registry;
}
async function writeFlow(name: string, yaml: Parameters<typeof serializeFlow>[0]): Promise<void> {
const dir = path.join(tmpDir, ".argent", "flows");
await fs.mkdir(dir, { recursive: true });
await fs.writeFile(path.join(dir, `${name}.yaml`), serializeFlow(yaml), "utf8");
}
function asRun(r: FlowRunResult | { notice: string }): FlowRunResult {
if (!("steps" in r)) throw new Error(`expected a run result, got notice: ${r.notice}`);
return r;
}
async function run(
name: string,
device = DEVICE
): Promise<FlowRunResult & { calls: Array<{ tool: string; args: Record<string, unknown> }> }> {
const calls: Array<{ tool: string; args: Record<string, unknown> }> = [];
const tool = createRunFlowTool(mockRegistry(calls));
const result = asRun(await tool.execute({}, { name, project_root: tmpDir, device }));
return Object.assign(result, { calls });
}
beforeEach(async () => {
tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "flow-rotate-"));
currentTree = () => screen([]);
currentScreen = () => ({ width: SCREEN_W, height: SCREEN_H });
});
afterEach(async () => {
await fs.rm(tmpDir, { recursive: true, force: true });
});
describe("rotate: parse/serialize", () => {
it("round-trips the by-only and on+by spellings", () => {
const flow = {
executionPrerequisite: "",
steps: [
{ kind: "rotate" as const, by: 90 },
{ kind: "rotate" as const, selector: { text: "Map", loose: true }, by: -30 },
{ kind: "rotate" as const, selector: { identifier: "map" }, by: 720 },
],
};
const reparsed = parseFlow(serializeFlow(flow)).steps;
expect(reparsed).toEqual(flow.steps);
});
it("emits `on`, then `by`", () => {
const yaml = serializeFlow({
executionPrerequisite: "",
steps: [
{ kind: "rotate", selector: { text: "Map", loose: true }, by: 90 },
{ kind: "rotate", by: -45 },
],
});
expect(yaml).toMatch(/rotate:\n\s+on: Map\n\s+by: 90\n/);
expect(yaml).toMatch(/rotate:\n\s+by: -45\n/);
});
it("parses the options map with the usual selector sugar", () => {
const steps = parseFlow(
"steps:\n" +
' - rotate: { on: "Map", by: 90 }\n' +
" - rotate: { on: { id: map }, by: -30 }\n" +
" - rotate: { by: 720 }\n"
).steps;
expect(steps).toEqual([
{ kind: "rotate", selector: { text: "Map", loose: true }, by: 90 },
{ kind: "rotate", selector: { identifier: "map" }, by: -30 },
{ kind: "rotate", by: 720 },
]);
});
it("rejects a non-map body with the options-map hint, not a by type error", () => {
for (const bad of ["90", '"Map"', "null"]) {
let message = "";
try {
parseFlow(`steps:\n - rotate: ${bad}\n`);
} catch (err) {
message = err instanceof Error ? err.message : String(err);
}
expect(message).toMatch(/rotate takes an options map/);
expect(message).not.toMatch(/by must be/);
}
});
it("rejects top-level selector fields with the nested-selector hint", () => {
expect(() => parseFlow('steps:\n - rotate: { text: "Map", by: 90 }\n')).toThrow(
/takes a nested selector/i
);
expect(() => parseFlow("steps:\n - rotate: { id: map, by: 90 }\n")).toThrow(
/takes a nested selector/i
);
});
it("rejects unknown keys with a typo suggestion", () => {
expect(() => parseFlow("steps:\n - rotate: { on: Map, by: 90, angle: 45 }\n")).toThrow(
/unknown key `angle`.*allowed keys: on, by/i
);
expect(() => parseFlow("steps:\n - rotate: { on: Map, byy: 90 }\n")).toThrow(
/did you mean `by`/i
);
});
it("rejects `duration` as an unknown key — the pace is fixed", () => {
expect(() => parseFlow("steps:\n - rotate: { by: 90, duration: 1200 }\n")).toThrow(
/unknown key `duration`.*allowed keys: on, by/i
);
});
it("rejects invalid angles", () => {
for (const bad of ["0", ".inf", '"90"']) {
expect(() => parseFlow(`steps:\n - rotate: { on: Map, by: ${bad} }\n`)).toThrow(
/rotate\.by must be a finite non-zero number of degrees \(\+CW, −CCW\)/
);
}
expect(() => parseFlow("steps:\n - rotate: { on: Map }\n")).toThrow(
/rotate\.by must be a finite non-zero number/
);
});
it("accepts angles up to ±3000° and rejects anything beyond, naming the limit", () => {
for (const ok of ["3000", "-3000"]) {
expect(() => parseFlow(`steps:\n - rotate: { by: ${ok} }\n`)).not.toThrow();
}
for (const bad of ["3000.001", "-3001"]) {
expect(() => parseFlow(`steps:\n - rotate: { by: ${bad} }\n`)).toThrow(
/rotate\.by must be within ±3000°/
);
}
});
});
describe("rotate: execution", () => {
it("dispatches one physically circular gesture at the element center with the derived duration", async () => {
// Center x ≠ y (0.4, 0.6) so a transposed centerX/centerY dispatch fails here.
currentTree = () =>
screen([n({ label: "Map", frame: { x: 0.1, y: 0.4, width: 0.6, height: 0.4 } })]);
await writeFlow("rotate-map", {
executionPrerequisite: "",
steps: [{ kind: "rotate", selector: { text: "Map", loose: true }, by: 90 }],
});
const result = await run("rotate-map");
expect(result.ok).toBe(true);
expect(result.calls).toHaveLength(1);
const { tool, args } = result.calls[0]!;
expect(tool).toBe("gesture-rotate");
expect(args.udid).toBe(DEVICE);
expect(args.centerX).toBeCloseTo(0.4, 9);
expect(args.centerY).toBeCloseTo(0.6, 9);
// Physical circle: one radius spelled in both normalizations, no ellipse.
expect((args.radiusX as number) * SCREEN_W).toBeCloseTo((args.radiusY as number) * SCREEN_H, 9);
expect(args).not.toHaveProperty("radius");
// +by = clockwise: endAngle − startAngle is exactly the requested delta.
expect((args.endAngle as number) - (args.startAngle as number)).toBe(90);
expect(args.durationMs).toBe(300); // ~90°/300ms
});
it("defaults to the screen center when no selector is given", async () => {
await writeFlow("rotate-center", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: -90 }],
});
const result = await run("rotate-center");
expect(result.ok).toBe(true);
const args = result.calls[0]!.args;
expect(args.centerX).toBeCloseTo(0.5, 9);
expect(args.centerY).toBeCloseTo(0.5, 9);
// Portrait phone: horizontal Down points would ride the side guards, so
// the fingers start on the vertical axis.
expect(args.startAngle).toBe(90);
expect((args.endAngle as number) - (args.startAngle as number)).toBe(-90);
expect((args.radiusX as number) * SCREEN_W).toBeCloseTo((args.radiusY as number) * SCREEN_H, 9);
});
it("derives every duration from the angle at the fixed pace", async () => {
await writeFlow("rotate-durations", {
executionPrerequisite: "",
steps: [
{ kind: "rotate", by: 720 },
{ kind: "rotate", by: -720 },
{ kind: "rotate", by: 3000 },
],
});
const result = await run("rotate-durations");
expect(result.ok).toBe(true);
expect(result.calls.map((c) => c.args.durationMs)).toEqual([2400, 2400, 10000]);
});
it("falls back to the legacy single-radius ellipse when the source reports no dimensions", async () => {
currentScreen = () => undefined;
await writeFlow("rotate-no-dims", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 90 }],
});
const result = await run("rotate-no-dims");
expect(result.ok).toBe(true);
const args = result.calls[0]!.args;
expect(args.radius).toBeGreaterThan(0);
expect(args).not.toHaveProperty("radiusX");
expect(args).not.toHaveProperty("radiusY");
});
it("still rotates (ellipse fallback) when the tree is unreadable and no selector is given", async () => {
currentTree = () => {
throw new Error("native devtools is unavailable");
};
await writeFlow("rotate-blind", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 45 }],
});
const result = await run("rotate-blind");
expect(result.ok).toBe(true);
const args = result.calls[0]!.args;
expect(args.radius).toBeGreaterThan(0);
expect(args.durationMs).toBe(300); // max(1, 45/90) floors at one 90° unit
// Generous budget: this is the run's first settle, so it pays the whole
// window before giving up and dispatching anyway.
}, 15_000);
it("fails only when no on-screen orbit radius exists — and never blames target size", async () => {
// Target centered inside the 2% screen inset: no circle fits around it.
currentTree = () =>
screen([n({ label: "Edge", frame: { x: 0, y: 0.4, width: 0.02, height: 0.2 } })]);
await writeFlow("rotate-no-room", {
executionPrerequisite: "",
steps: [{ kind: "rotate", selector: { text: "Edge", loose: true }, by: 90 }],
});
const result = await run("rotate-no-room");
expect(result.ok).toBe(false);
expect(result.steps[0]).toMatchObject({ kind: "rotate", status: "fail" });
expect(result.steps[0].reason).toMatch(/no on-screen orbit radius/);
expect(result.steps[0].reason).not.toMatch(/small|tiny/i);
expect(result.calls).toHaveLength(0);
});
it("fails with the scroll-to hint when the target never appears", async () => {
currentTree = () => screen([]);
await writeFlow("rotate-missing", {
executionPrerequisite: "",
steps: [{ kind: "rotate", selector: { text: "Map", loose: true }, by: 90 }],
});
const result = await run("rotate-missing");
expect(result.ok).toBe(false);
expect(result.steps[0]).toMatchObject({ kind: "rotate", status: "fail" });
expect(result.steps[0].reason).toMatch(/add a scroll-to step/i);
expect(result.calls).toHaveLength(0);
}, 15000);
});
describe("rotate: gating", () => {
it("rejects rotate on vega with the touch-directive message shape", async () => {
await writeFlow("rotate-vega", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 90 }],
});
const result = await run("rotate-vega", "amazon-4a27df03c9777152");
expect(result.steps[0]).toMatchObject({ kind: "rotate", status: "fail" });
expect(result.steps[0].reason).toMatch(
/rotate is a touch directive and Vega is remote-driven — move focus with `tool: tv-remote`/
);
expect(result.calls).toHaveLength(0);
});
it("rejects rotate on chromium for the missing rotate idiom, not a missing backend", async () => {
await writeFlow("rotate-chromium", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 90 }],
});
const result = await run("rotate-chromium", "chromium-cdp-9222");
expect(result.steps[0]).toMatchObject({ kind: "rotate", status: "fail" });
expect(result.steps[0].reason).toMatch(/no rotate-gesture idiom/);
expect(result.steps[0].reason).toMatch(/tap\/keyboard/);
expect(result.steps[0].reason).not.toMatch(/no backend/i);
expect(result.calls).toHaveLength(0);
});
});
describe("rotate: abort", () => {
it("aborts inside the pre-gesture settle: aborted outcome, no gesture", async () => {
const controller = new AbortController();
// Cancel on read 2, the settle's second read (identical trees converge
// there). Read 1 is ambiguous - without the settle it is the aspect read,
// which aborts just as cleanly; only read 2 exists when the settle does.
let reads = 0;
currentTree = () => {
reads += 1;
if (reads >= 2) controller.abort();
return screen([]);
};
const calls: Array<{ tool: string; args: Record<string, unknown> }> = [];
await writeFlow("rotate-abort", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 720 }],
});
const result = asRun(
await createRunFlowTool(mockRegistry(calls)).execute(
{},
{ name: "rotate-abort", project_root: tmpDir, device: DEVICE },
{ signal: controller.signal } as never
)
);
expect(reads).toBe(2);
expect(result.steps.map((s) => `${s.kind}:${s.status}`)).toEqual(["rotate:skip"]);
expect(result.steps[0].reason).toBe("run aborted");
expect(calls).toHaveLength(0);
});
it("aborts during the aspect read: aborted outcome, no gesture", async () => {
const controller = new AbortController();
// Reads 1 and 2 are the settle (identical trees converge immediately);
// read 3 is the aspect read. Cancelling there covers the window between
// the settle and the dispatch, which the settle's own abort cannot reach.
let reads = 0;
currentTree = () => {
reads += 1;
if (reads >= 3) controller.abort();
return screen([]);
};
const calls: Array<{ tool: string; args: Record<string, unknown> }> = [];
await writeFlow("rotate-abort-aspect", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 720 }],
});
const result = asRun(
await createRunFlowTool(mockRegistry(calls)).execute(
{},
{ name: "rotate-abort-aspect", project_root: tmpDir, device: DEVICE },
{ signal: controller.signal } as never
)
);
expect(reads).toBe(3);
expect(result.steps.map((s) => `${s.kind}:${s.status}`)).toEqual(["rotate:skip"]);
expect(result.steps[0].reason).toBe("run aborted");
expect(calls).toHaveLength(0);
});
it("maps a mid-gesture abort rejection to the aborted skip, not a failure", async () => {
const controller = new AbortController();
// The tool now rejects when cancelled mid-gesture: abort, then reject, from
// inside the dispatch — the flow must read the signal, not the message.
const registry = {
invokeTool: vi.fn(async (id: string) => {
if (id === "list-devices") return { devices: [] };
if (id === "gesture-rotate") {
controller.abort();
throw new Error("gesture-rotate aborted — cancelled mid-gesture after 3 of 31 frames");
}
return { ok: true };
}),
getTool: vi.fn(() => ({ inputSchema: { properties: { udid: {} } } })),
} as unknown as Registry;
await writeFlow("rotate-mid-abort", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 90 }],
});
const result = asRun(
await createRunFlowTool(registry).execute(
{},
{ name: "rotate-mid-abort", project_root: tmpDir, device: DEVICE },
{ signal: controller.signal } as never
)
);
expect(result.steps.map((s) => `${s.kind}:${s.status}`)).toEqual(["rotate:skip"]);
expect(result.steps[0].reason).toBe("run aborted");
});
it("propagates a dispatch rejection as a real error when the run is not aborted", async () => {
const registry = {
invokeTool: vi.fn(async (id: string) => {
if (id === "list-devices") return { devices: [] };
if (id === "gesture-rotate") throw new Error("simulator-server unreachable");
return { ok: true };
}),
getTool: vi.fn(() => ({ inputSchema: { properties: { udid: {} } } })),
} as unknown as Registry;
await writeFlow("rotate-dispatch-error", {
executionPrerequisite: "",
steps: [{ kind: "rotate", by: 90 }],
});
const result = asRun(
await createRunFlowTool(registry).execute(
{},
{ name: "rotate-dispatch-error", project_root: tmpDir, device: DEVICE }
)
);
expect(result.steps[0]).toMatchObject({ kind: "rotate", status: "error" });
expect(result.steps[0].reason).toMatch(/simulator-server unreachable/);
});
});