Files
software-mansion__argent/packages/tool-server/test/flows/flow-idle-condition.test.ts
Hubert Gancarczyk 96ea94f87a feat(flow): add an idle condition that waits for the screen to stop moving (#728)
> Stacked on #574. Replaces the route-fingerprint approach that #727 and
#729 carried; both are closed.
>
> Rebased off the #715-#726 chain so it can merge on #574 alone. That
chain owns `DirectiveOutcome.warning` / `StepReport.warning` and the
`indeterminate` scoring rule; the small amount of each that this
condition needs is carried here, scoped to `idle`, and will conflict
textually with #724/#725 for whichever lands second.

## One check a selector cannot express

```yaml
- await: { idle: true }
```

Has the screen stopped moving? Flows were substituting fixed `wait:`
steps for it, which either wait too long on every run or too little on
the run that mattered.

## Why it reads pixels as well as the tree

Each signal is blind to what the other sees.

The tree cannot see presentation-layer motion. An iOS push or modal
dismissal commits its hierarchy up front and then animates a layer for a
few hundred milliseconds; a cross-fade or a scrim moves no node at all.
Measured on a real Android emulator: a page animating a large element
continuously reads as a single static `WebView` node, and the existing
`await-screen-idle` tool called it settled in **465 ms, five times out
of five**. On a deterministic Chromium page whose DOM is byte-identical
across the animation, that tool returned `settled: true` in **407 ms**
while this condition failed 8/8.

Pixels alone would not do either: they cannot see a tree still churning
behind an unchanged surface, and anything animated forever (a video, a
shimmer) would make a pixel-only settle unsatisfiable on a screen the
tree calls ready.

Unlike that tool, this cannot be silently stepped over. Readiness is not
an acceptance criterion — the verdict belongs to the identity and
outcome checks around it — so a screen that never settles still
**passes**, carrying a `warning` that says what the green actually
bought. That is what makes it safe to persist in a flow: it reports on
every run without ever failing one over a video or a shimmer. The only
thing that stops a run is a tree source that cannot be read.

## The rule the design follows

The absence of evidence is never evidence.

- A capture that did not arrive cannot complete the hold.
- Neither can a single agreeing pair. Every animation that reverses has
a turning point, and two samples straddling it come back identical while
the screen is moving. On a live 3 s cross-fade that passed a
default-shaped step on roughly **one run in three**; settling now takes
two consecutive still intervals, so `minStableMs: 0` still means three
reads.
- A round is never started without the budget to observe it with. A
round begun with nothing left neither captures nor reads, and both
absences were otherwise recorded as facts about the device.
- No verdict comes from a latch. A screen that settles and then moves
again, or goes blank, has not settled.

## `timeout:` is a real bound

No describe path takes an abort signal, so the tree read is raced
against what is left of the budget. Without that, a wedged ViewInspector
RPC ran 2.25 s past an 8000 ms budget, and a busy Android screen ran
5.4-6.0 s against a 3000 ms one. A read that runs out of step budget is
the step ending, not the source failing, and only the latter is reported
as an environment problem.

## Failure modes stay apart

They call for opposite responses. A screen that never stopped moving is
a verdict about the app. A tree that could not be read is not, and it
names the foreground check first, because a backgrounded app reads
identically to an uninstrumented one and relaunching is the wrong
repair. A run whose captures never produced a comparable pair still
settles on the tree and says so in a warning, rather than passing off
half a proof as the whole one.

## Capture routing and tolerance

Captures route exactly as the `screenshot` tool routes them, so tvOS and
Vega, which have no simulator-server backend but are perfectly
screenshottable through `xcrun` and the emulator console, are covered
rather than written off.

The pixel comparison owns its tolerance rather than borrowing
`screenshot-diff`'s. That one holds a baseline stored across sessions,
machines and OS versions against a live capture and must absorb real
drift; this one holds two captures one poll apart from a single session,
where a static screen reads back byte-identical (measured: zero changed
pixels across five consecutive pairs on an idle simulator). The margin
is load-bearing, because uniform change is all-or-nothing: at the
baseline tolerance any cross-fade slower than ~2 s counted zero changed
pixels and read as settled mid-animation.

## Not a screen check

A dropped tap leaves the source screen perfectly idle. It belongs after
the element check that names the destination, never instead of one. #730
documents the pairing.

## Verification

- 3307 tests pass; typecheck, `typecheck:tests`, lint and prettier
clean. Eleven mutations applied to the source, all eleven caught by
tests.
- Driven end to end against a real iOS simulator, a real Android
emulator, and a Chromium app through a tool-server built from this
branch: static screens pass with no warning (proving captures ran),
animating screens pass carrying the warning that names the motion, and
the pre-existing tool disagrees exactly where it should.

---------

Co-authored-by: Hubert Gancarczyk <claude-hubert.gancarczyk@swmansion.com>
2026-08-10 11:50:16 +02:00

229 lines
10 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { parseFlow, serializeFlow, type FlowStep } from "../../src/tools/flows/flow-utils";
// `idle` is the one condition that takes no selector. It shares the `await:`
// key with the selector conditions, so the parse/serialize round trip and the
// mutual exclusion between the two families are the load-bearing behaviors.
const flow = (steps: string): string => `executionPrerequisite: ""\nsteps:\n${steps}`;
function parseSteps(steps: string): FlowStep[] {
return parseFlow(flow(steps)).steps;
}
/** A flow's steps survive serialize → parse unchanged (canonical spelling). */
function expectRoundTrip(steps: string): FlowStep[] {
const parsed = parseSteps(steps);
expect(parseFlow(serializeFlow({ executionPrerequisite: "", steps: parsed })).steps).toEqual(
parsed
);
return parsed;
}
describe("await { idle }", () => {
it("parses the readiness gate and round-trips its minimal spelling", () => {
const steps = expectRoundTrip(` - await: { idle: true }\n`);
expect(steps).toEqual([{ kind: "idle" }]);
// Minimal in, minimal out — no defaults materialize into the file.
expect(serializeFlow({ executionPrerequisite: "", steps })).toContain(
"await:\n idle: true"
);
});
it("carries the optional hold and timeout", () => {
expect(expectRoundTrip(` - await: { idle: true, stableFor: 400, timeout: 9000 }\n`)).toEqual([
{ kind: "idle", stableFor: 400, timeout: 9000 },
]);
});
it("has no assert form — waiting is the whole point of the check", () => {
expect(() => parseSteps(` - assert: { idle: true }\n`)).toThrow(/idle has no assert form/);
});
it("takes only `true` — there is no useful 'prove the screen is moving'", () => {
expect(() => parseSteps(` - await: { idle: false }\n`)).toThrow(/idle takes only/);
});
it("bounds stableFor", () => {
expect(() => parseSteps(` - await: { idle: true, stableFor: -1 }\n`)).toThrow(
/idle.stableFor/
);
expect(() => parseSteps(` - await: { idle: true, stableFor: 1.5 }\n`)).toThrow(
/idle.stableFor/
);
// And from above, so a hold written in the wrong unit (seconds, a pasted
// timestamp) is rejected as a number rather than becoming a gate no run
// can pass. The ceiling is ten minutes; a `timeout` wide enough to contain
// it is what the case below checks separately.
expect(() => parseSteps(` - await: { idle: true, stableFor: 600001 }\n`)).toThrow(
/between 0 and 600000/
);
expect(parseSteps(` - await: { idle: true, stableFor: 600000, timeout: 600600 }\n`)).toEqual([
{ kind: "idle", stableFor: 600000, timeout: 600600 },
]);
});
// A wait that cannot contain the settle it asks for is a gate that fails on
// every run — and fails blaming the app, which is the one thing it is not
// evidence about. Caught at parse, deviceless, rather than against a live
// screen.
//
// The settle spans 400ms — three reads over two 200ms polls — and the hold is
// counted ACROSS those polls, not after them, so what the wait has to contain
// is the longer of the two plus the budget the closing round has to start
// with. Summing them instead rejected steps the runner satisfies: measured
// through the runner, `stableFor: 800` settles in ~820ms and the default hold
// in ~411ms, against a demand of 1400ms and 850ms.
it("rejects a wait that could never contain the settle it asks for", () => {
expect(() => parseSteps(` - await: { idle: true, timeout: 500, stableFor: 1000 }\n`)).toThrow(
/idle needs a timeout of at least 1200ms to hold still for 1000ms/
);
// With no explicit hold the DEFAULT is what has to fit — the spelling that
// slipped through, since leaving `stableFor` out was the way past the
// check that only looked at a written-out one.
expect(() => parseSteps(` - await: { idle: true, timeout: 100 }\n`)).toThrow(
/idle needs a timeout of at least 600ms to hold still for the default 250ms/
);
// Which is the same step as writing the default out, so it is rejected the
// same way.
expect(() => parseSteps(` - await: { idle: true, timeout: 100, stableFor: 250 }\n`)).toThrow(
/idle needs a timeout of at least 600ms/
);
// With no explicit timeout the default is what the hold has to fit inside.
expect(() => parseSteps(` - await: { idle: true, stableFor: 9000 }\n`)).toThrow(
/idle needs a timeout of at least 9200ms/
);
// Under the span, the span is the floor and the hold does not add to it —
// both of these ask for the same 600ms.
expect(() => parseSteps(` - await: { idle: true, timeout: 599, stableFor: 0 }\n`)).toThrow(
/idle needs a timeout of at least 600ms/
);
expect(() => parseSteps(` - await: { idle: true, timeout: 599, stableFor: 400 }\n`)).toThrow(
/idle needs a timeout of at least 600ms/
);
// The boundary itself is legal on both sides.
expect(parseSteps(` - await: { idle: true, timeout: 700, stableFor: 500 }\n`)).toEqual([
{ kind: "idle", timeout: 700, stableFor: 500 },
]);
expect(() => parseSteps(` - await: { idle: true, timeout: 699, stableFor: 500 }\n`)).toThrow(
/idle needs a timeout of at least 700ms/
);
});
// The floor is not merely smaller — it is the one the runner needs. A hold
// the parser calls impossible must not be one a still screen serves, which is
// what the additive sum produced: these are the two cases measured above.
it("accepts the smallest wait the runner can actually settle in", () => {
expect(parseSteps(` - await: { idle: true, timeout: 1000, stableFor: 800 }\n`)).toEqual([
{ kind: "idle", timeout: 1000, stableFor: 800 },
]);
expect(parseSteps(` - await: { idle: true, timeout: 600 }\n`)).toEqual([
{ kind: "idle", timeout: 600 },
]);
});
// The one near-miss the docs actively produce: every other condition is
// written with a selector beside it, so `await:` comes along for free, while
// this one reads like a directive of its own.
it("names the spelling when `idle` is written as a step of its own", () => {
expect(() => parseSteps(` - idle: true\n`)).toThrow(
/idle is a condition, not a step kind — write it as `await: \{ idle: true \}`/
);
});
// The hold is validated as an integer while the timeout is not, so a
// fractional timeout used to be turned into a bound for the hold and asked
// for "an integer between 0 and -0.5". Nothing is derived from it now: the
// two are checked against each other as a sum, which has an answer whatever
// the timeout is.
it("rejects a timeout too small to settle in without inventing a range", () => {
const parse = (): FlowStep[] =>
parseSteps(` - await: { idle: true, timeout: 0.5, stableFor: 0 }\n`);
expect(parse).toThrow(/idle needs a timeout of at least 600ms/);
expect(parse).not.toThrow(/between 0 and -/);
});
it("rejects a non-positive timeout", () => {
expect(() => parseSteps(` - await: { idle: true, timeout: 0 }\n`)).toThrow(/await.timeout/);
expect(() => parseSteps(` - await: { idle: true, timeout: "soon" }\n`)).toThrow(
/await.timeout/
);
});
});
describe("condition families are mutually exclusive", () => {
it("rejects mixing a selector condition with the readiness one", () => {
expect(() => parseSteps(` - await: { idle: true, visible: { id: x } }\n`)).toThrow(
/mixes `idle` with `visible`/
);
});
// The same mix under `assert` used to cost two round trips: it reported the
// mixing first, and the split the author was told to write —
// `assert: { idle: true }` — is not valid either. One error has to end it.
it("sends an assert body that mixes the two straight to the form it needs", () => {
let message = "";
try {
parseSteps(` - assert: { idle: true, visible: { id: x } }\n`);
} catch (err) {
message = err instanceof Error ? err.message : String(err);
}
expect(message).toContain("idle has no assert form");
expect(message).toContain("await: { idle: true }");
expect(message).toContain("`visible`");
// And what it tells the author to write must itself parse.
expect(() =>
parseSteps(` - await: { idle: true }\n - assert: { visible: { id: x } }\n`)
).not.toThrow();
});
it("rejects a stray key rather than ignoring it", () => {
expect(() => parseSteps(` - await: { idle: true, settleMs: 500 }\n`)).toThrow(/settleMs/);
});
// A typo next to an `idle:` gate used to be told that `idle` itself was not a
// legal key — the parser lists what the AUTHOR may write, which is not the
// same set as what its selector-condition branch parses.
it("offers idle when an await names no legal condition", () => {
expect(() => parseSteps(` - await: { visble: { id: home } }\n`)).toThrow(
/await needs exactly one condition key \(exists, visible, hidden, text, idle\)/
);
// Same list when the body isn't a condition map at all.
expect(() => parseSteps(` - await: visible\n`)).toThrow(
/await needs a condition \(exists, visible, hidden, text, idle\)/
);
});
it("does not offer it to assert or to a `when:` guard, which have no readiness form", () => {
const assertMiss = (): FlowStep[] => parseSteps(` - assert: { visble: { id: home } }\n`);
expect(assertMiss).toThrow(
/assert needs exactly one condition key \(exists, visible, hidden, text\)/
);
expect(assertMiss).not.toThrow(/idle/);
const guard = (body: string) => (): FlowStep[] =>
parseSteps(` - when: ${body}\n steps:\n - echo: guarded\n`);
// A stray key carrying neither substring, so the rejected entry echoed
// back into the message cannot satisfy the negative assertion.
const stray = guard("{ visble: { id: home } }");
expect(stray).toThrow(
/when needs exactly one condition key \(exists, visible, hidden, text, platform\)/
);
expect(stray).not.toThrow(/idle/);
// And `idle` itself is not a guard — which is said outright, the way the
// assert form is, rather than left to be inferred from a list it is
// missing from.
expect(guard("{ idle: true }")).toThrow(
/when has no idle form .* Put `await: \{ idle: true \}` before the block/
);
});
it("leaves the selector conditions untouched", () => {
expect(parseSteps(` - await: { visible: { id: home-screen } }\n`)).toEqual([
{ kind: "await", condition: "visible", selector: { identifier: "home-screen" } },
]);
});
});