mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
96ea94f87a
> Stacked on #574. Replaces the route-fingerprint approach that #727 and #729 carried; both are closed. > > Rebased off the #715-#726 chain so it can merge on #574 alone. That chain owns `DirectiveOutcome.warning` / `StepReport.warning` and the `indeterminate` scoring rule; the small amount of each that this condition needs is carried here, scoped to `idle`, and will conflict textually with #724/#725 for whichever lands second. ## One check a selector cannot express ```yaml - await: { idle: true } ``` Has the screen stopped moving? Flows were substituting fixed `wait:` steps for it, which either wait too long on every run or too little on the run that mattered. ## Why it reads pixels as well as the tree Each signal is blind to what the other sees. The tree cannot see presentation-layer motion. An iOS push or modal dismissal commits its hierarchy up front and then animates a layer for a few hundred milliseconds; a cross-fade or a scrim moves no node at all. Measured on a real Android emulator: a page animating a large element continuously reads as a single static `WebView` node, and the existing `await-screen-idle` tool called it settled in **465 ms, five times out of five**. On a deterministic Chromium page whose DOM is byte-identical across the animation, that tool returned `settled: true` in **407 ms** while this condition failed 8/8. Pixels alone would not do either: they cannot see a tree still churning behind an unchanged surface, and anything animated forever (a video, a shimmer) would make a pixel-only settle unsatisfiable on a screen the tree calls ready. Unlike that tool, this cannot be silently stepped over. Readiness is not an acceptance criterion — the verdict belongs to the identity and outcome checks around it — so a screen that never settles still **passes**, carrying a `warning` that says what the green actually bought. That is what makes it safe to persist in a flow: it reports on every run without ever failing one over a video or a shimmer. The only thing that stops a run is a tree source that cannot be read. ## The rule the design follows The absence of evidence is never evidence. - A capture that did not arrive cannot complete the hold. - Neither can a single agreeing pair. Every animation that reverses has a turning point, and two samples straddling it come back identical while the screen is moving. On a live 3 s cross-fade that passed a default-shaped step on roughly **one run in three**; settling now takes two consecutive still intervals, so `minStableMs: 0` still means three reads. - A round is never started without the budget to observe it with. A round begun with nothing left neither captures nor reads, and both absences were otherwise recorded as facts about the device. - No verdict comes from a latch. A screen that settles and then moves again, or goes blank, has not settled. ## `timeout:` is a real bound No describe path takes an abort signal, so the tree read is raced against what is left of the budget. Without that, a wedged ViewInspector RPC ran 2.25 s past an 8000 ms budget, and a busy Android screen ran 5.4-6.0 s against a 3000 ms one. A read that runs out of step budget is the step ending, not the source failing, and only the latter is reported as an environment problem. ## Failure modes stay apart They call for opposite responses. A screen that never stopped moving is a verdict about the app. A tree that could not be read is not, and it names the foreground check first, because a backgrounded app reads identically to an uninstrumented one and relaunching is the wrong repair. A run whose captures never produced a comparable pair still settles on the tree and says so in a warning, rather than passing off half a proof as the whole one. ## Capture routing and tolerance Captures route exactly as the `screenshot` tool routes them, so tvOS and Vega, which have no simulator-server backend but are perfectly screenshottable through `xcrun` and the emulator console, are covered rather than written off. The pixel comparison owns its tolerance rather than borrowing `screenshot-diff`'s. That one holds a baseline stored across sessions, machines and OS versions against a live capture and must absorb real drift; this one holds two captures one poll apart from a single session, where a static screen reads back byte-identical (measured: zero changed pixels across five consecutive pairs on an idle simulator). The margin is load-bearing, because uniform change is all-or-nothing: at the baseline tolerance any cross-fade slower than ~2 s counted zero changed pixels and read as settled mid-animation. ## Not a screen check A dropped tap leaves the source screen perfectly idle. It belongs after the element check that names the destination, never instead of one. #730 documents the pairing. ## Verification - 3307 tests pass; typecheck, `typecheck:tests`, lint and prettier clean. Eleven mutations applied to the source, all eleven caught by tests. - Driven end to end against a real iOS simulator, a real Android emulator, and a Chromium app through a tool-server built from this branch: static screens pass with no warning (proving captures ran), animating screens pass carrying the warning that names the motion, and the pre-existing tool disagrees exactly where it should. --------- Co-authored-by: Hubert Gancarczyk <claude-hubert.gancarczyk@swmansion.com>
229 lines
10 KiB
TypeScript
229 lines
10 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { parseFlow, serializeFlow, type FlowStep } from "../../src/tools/flows/flow-utils";
|
|
|
|
// `idle` is the one condition that takes no selector. It shares the `await:`
|
|
// key with the selector conditions, so the parse/serialize round trip and the
|
|
// mutual exclusion between the two families are the load-bearing behaviors.
|
|
|
|
const flow = (steps: string): string => `executionPrerequisite: ""\nsteps:\n${steps}`;
|
|
|
|
function parseSteps(steps: string): FlowStep[] {
|
|
return parseFlow(flow(steps)).steps;
|
|
}
|
|
|
|
/** A flow's steps survive serialize → parse unchanged (canonical spelling). */
|
|
function expectRoundTrip(steps: string): FlowStep[] {
|
|
const parsed = parseSteps(steps);
|
|
expect(parseFlow(serializeFlow({ executionPrerequisite: "", steps: parsed })).steps).toEqual(
|
|
parsed
|
|
);
|
|
return parsed;
|
|
}
|
|
|
|
describe("await { idle }", () => {
|
|
it("parses the readiness gate and round-trips its minimal spelling", () => {
|
|
const steps = expectRoundTrip(` - await: { idle: true }\n`);
|
|
expect(steps).toEqual([{ kind: "idle" }]);
|
|
// Minimal in, minimal out — no defaults materialize into the file.
|
|
expect(serializeFlow({ executionPrerequisite: "", steps })).toContain(
|
|
"await:\n idle: true"
|
|
);
|
|
});
|
|
|
|
it("carries the optional hold and timeout", () => {
|
|
expect(expectRoundTrip(` - await: { idle: true, stableFor: 400, timeout: 9000 }\n`)).toEqual([
|
|
{ kind: "idle", stableFor: 400, timeout: 9000 },
|
|
]);
|
|
});
|
|
|
|
it("has no assert form — waiting is the whole point of the check", () => {
|
|
expect(() => parseSteps(` - assert: { idle: true }\n`)).toThrow(/idle has no assert form/);
|
|
});
|
|
|
|
it("takes only `true` — there is no useful 'prove the screen is moving'", () => {
|
|
expect(() => parseSteps(` - await: { idle: false }\n`)).toThrow(/idle takes only/);
|
|
});
|
|
|
|
it("bounds stableFor", () => {
|
|
expect(() => parseSteps(` - await: { idle: true, stableFor: -1 }\n`)).toThrow(
|
|
/idle.stableFor/
|
|
);
|
|
expect(() => parseSteps(` - await: { idle: true, stableFor: 1.5 }\n`)).toThrow(
|
|
/idle.stableFor/
|
|
);
|
|
// And from above, so a hold written in the wrong unit (seconds, a pasted
|
|
// timestamp) is rejected as a number rather than becoming a gate no run
|
|
// can pass. The ceiling is ten minutes; a `timeout` wide enough to contain
|
|
// it is what the case below checks separately.
|
|
expect(() => parseSteps(` - await: { idle: true, stableFor: 600001 }\n`)).toThrow(
|
|
/between 0 and 600000/
|
|
);
|
|
expect(parseSteps(` - await: { idle: true, stableFor: 600000, timeout: 600600 }\n`)).toEqual([
|
|
{ kind: "idle", stableFor: 600000, timeout: 600600 },
|
|
]);
|
|
});
|
|
|
|
// A wait that cannot contain the settle it asks for is a gate that fails on
|
|
// every run — and fails blaming the app, which is the one thing it is not
|
|
// evidence about. Caught at parse, deviceless, rather than against a live
|
|
// screen.
|
|
//
|
|
// The settle spans 400ms — three reads over two 200ms polls — and the hold is
|
|
// counted ACROSS those polls, not after them, so what the wait has to contain
|
|
// is the longer of the two plus the budget the closing round has to start
|
|
// with. Summing them instead rejected steps the runner satisfies: measured
|
|
// through the runner, `stableFor: 800` settles in ~820ms and the default hold
|
|
// in ~411ms, against a demand of 1400ms and 850ms.
|
|
it("rejects a wait that could never contain the settle it asks for", () => {
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: 500, stableFor: 1000 }\n`)).toThrow(
|
|
/idle needs a timeout of at least 1200ms to hold still for 1000ms/
|
|
);
|
|
// With no explicit hold the DEFAULT is what has to fit — the spelling that
|
|
// slipped through, since leaving `stableFor` out was the way past the
|
|
// check that only looked at a written-out one.
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: 100 }\n`)).toThrow(
|
|
/idle needs a timeout of at least 600ms to hold still for the default 250ms/
|
|
);
|
|
// Which is the same step as writing the default out, so it is rejected the
|
|
// same way.
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: 100, stableFor: 250 }\n`)).toThrow(
|
|
/idle needs a timeout of at least 600ms/
|
|
);
|
|
// With no explicit timeout the default is what the hold has to fit inside.
|
|
expect(() => parseSteps(` - await: { idle: true, stableFor: 9000 }\n`)).toThrow(
|
|
/idle needs a timeout of at least 9200ms/
|
|
);
|
|
// Under the span, the span is the floor and the hold does not add to it —
|
|
// both of these ask for the same 600ms.
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: 599, stableFor: 0 }\n`)).toThrow(
|
|
/idle needs a timeout of at least 600ms/
|
|
);
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: 599, stableFor: 400 }\n`)).toThrow(
|
|
/idle needs a timeout of at least 600ms/
|
|
);
|
|
// The boundary itself is legal on both sides.
|
|
expect(parseSteps(` - await: { idle: true, timeout: 700, stableFor: 500 }\n`)).toEqual([
|
|
{ kind: "idle", timeout: 700, stableFor: 500 },
|
|
]);
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: 699, stableFor: 500 }\n`)).toThrow(
|
|
/idle needs a timeout of at least 700ms/
|
|
);
|
|
});
|
|
|
|
// The floor is not merely smaller — it is the one the runner needs. A hold
|
|
// the parser calls impossible must not be one a still screen serves, which is
|
|
// what the additive sum produced: these are the two cases measured above.
|
|
it("accepts the smallest wait the runner can actually settle in", () => {
|
|
expect(parseSteps(` - await: { idle: true, timeout: 1000, stableFor: 800 }\n`)).toEqual([
|
|
{ kind: "idle", timeout: 1000, stableFor: 800 },
|
|
]);
|
|
expect(parseSteps(` - await: { idle: true, timeout: 600 }\n`)).toEqual([
|
|
{ kind: "idle", timeout: 600 },
|
|
]);
|
|
});
|
|
|
|
// The one near-miss the docs actively produce: every other condition is
|
|
// written with a selector beside it, so `await:` comes along for free, while
|
|
// this one reads like a directive of its own.
|
|
it("names the spelling when `idle` is written as a step of its own", () => {
|
|
expect(() => parseSteps(` - idle: true\n`)).toThrow(
|
|
/idle is a condition, not a step kind — write it as `await: \{ idle: true \}`/
|
|
);
|
|
});
|
|
|
|
// The hold is validated as an integer while the timeout is not, so a
|
|
// fractional timeout used to be turned into a bound for the hold and asked
|
|
// for "an integer between 0 and -0.5". Nothing is derived from it now: the
|
|
// two are checked against each other as a sum, which has an answer whatever
|
|
// the timeout is.
|
|
it("rejects a timeout too small to settle in without inventing a range", () => {
|
|
const parse = (): FlowStep[] =>
|
|
parseSteps(` - await: { idle: true, timeout: 0.5, stableFor: 0 }\n`);
|
|
expect(parse).toThrow(/idle needs a timeout of at least 600ms/);
|
|
expect(parse).not.toThrow(/between 0 and -/);
|
|
});
|
|
|
|
it("rejects a non-positive timeout", () => {
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: 0 }\n`)).toThrow(/await.timeout/);
|
|
expect(() => parseSteps(` - await: { idle: true, timeout: "soon" }\n`)).toThrow(
|
|
/await.timeout/
|
|
);
|
|
});
|
|
});
|
|
|
|
describe("condition families are mutually exclusive", () => {
|
|
it("rejects mixing a selector condition with the readiness one", () => {
|
|
expect(() => parseSteps(` - await: { idle: true, visible: { id: x } }\n`)).toThrow(
|
|
/mixes `idle` with `visible`/
|
|
);
|
|
});
|
|
|
|
// The same mix under `assert` used to cost two round trips: it reported the
|
|
// mixing first, and the split the author was told to write —
|
|
// `assert: { idle: true }` — is not valid either. One error has to end it.
|
|
it("sends an assert body that mixes the two straight to the form it needs", () => {
|
|
let message = "";
|
|
try {
|
|
parseSteps(` - assert: { idle: true, visible: { id: x } }\n`);
|
|
} catch (err) {
|
|
message = err instanceof Error ? err.message : String(err);
|
|
}
|
|
expect(message).toContain("idle has no assert form");
|
|
expect(message).toContain("await: { idle: true }");
|
|
expect(message).toContain("`visible`");
|
|
// And what it tells the author to write must itself parse.
|
|
expect(() =>
|
|
parseSteps(` - await: { idle: true }\n - assert: { visible: { id: x } }\n`)
|
|
).not.toThrow();
|
|
});
|
|
|
|
it("rejects a stray key rather than ignoring it", () => {
|
|
expect(() => parseSteps(` - await: { idle: true, settleMs: 500 }\n`)).toThrow(/settleMs/);
|
|
});
|
|
|
|
// A typo next to an `idle:` gate used to be told that `idle` itself was not a
|
|
// legal key — the parser lists what the AUTHOR may write, which is not the
|
|
// same set as what its selector-condition branch parses.
|
|
it("offers idle when an await names no legal condition", () => {
|
|
expect(() => parseSteps(` - await: { visble: { id: home } }\n`)).toThrow(
|
|
/await needs exactly one condition key \(exists, visible, hidden, text, idle\)/
|
|
);
|
|
// Same list when the body isn't a condition map at all.
|
|
expect(() => parseSteps(` - await: visible\n`)).toThrow(
|
|
/await needs a condition \(exists, visible, hidden, text, idle\)/
|
|
);
|
|
});
|
|
|
|
it("does not offer it to assert or to a `when:` guard, which have no readiness form", () => {
|
|
const assertMiss = (): FlowStep[] => parseSteps(` - assert: { visble: { id: home } }\n`);
|
|
expect(assertMiss).toThrow(
|
|
/assert needs exactly one condition key \(exists, visible, hidden, text\)/
|
|
);
|
|
expect(assertMiss).not.toThrow(/idle/);
|
|
|
|
const guard = (body: string) => (): FlowStep[] =>
|
|
parseSteps(` - when: ${body}\n steps:\n - echo: guarded\n`);
|
|
|
|
// A stray key carrying neither substring, so the rejected entry echoed
|
|
// back into the message cannot satisfy the negative assertion.
|
|
const stray = guard("{ visble: { id: home } }");
|
|
expect(stray).toThrow(
|
|
/when needs exactly one condition key \(exists, visible, hidden, text, platform\)/
|
|
);
|
|
expect(stray).not.toThrow(/idle/);
|
|
|
|
// And `idle` itself is not a guard — which is said outright, the way the
|
|
// assert form is, rather than left to be inferred from a list it is
|
|
// missing from.
|
|
expect(guard("{ idle: true }")).toThrow(
|
|
/when has no idle form .* Put `await: \{ idle: true \}` before the block/
|
|
);
|
|
});
|
|
|
|
it("leaves the selector conditions untouched", () => {
|
|
expect(parseSteps(` - await: { visible: { id: home-screen } }\n`)).toEqual([
|
|
{ kind: "await", condition: "visible", selector: { identifier: "home-screen" } },
|
|
]);
|
|
});
|
|
});
|