The motion oracle borrowed screenshot-diff's DEFAULT_THRESHOLD, a value
sized for a baseline held across sessions and machines and so obliged to
absorb real drift. This comparison holds two captures one poll apart
from a single live session, where a static screen reads back
byte-identical: both capture paths are lossless PNG and the downscale is
deterministic. The borrowed value put the gate at ~25.5 levels/channel,
which mattered because spatially uniform change is all-or-nothing —
every pixel moves by the same amount, so it clears the gate on all of
them or on none, and MOTION_FRACTION is never the deciding term. A
cross-fade slower than ~1.4s therefore counted zero pixels and reported
settled mid-animation, over a screen still painted and still
hit-testing; and being settled rather than timed-out, degradedReason
stayed silent while updateBaselines could adopt a mid-fade frame as
golden.
0.03 (~7.6 levels/channel) keeps a margin over the +5/channel drift the
tests treat as the noise floor and moves that cliff out ~3.3x, to ~4.5s
at full contrast. It stays a rate floor rather than a duration floor, so
the comment now names the gate as the only term bounding what is left.
This supersedes the parity pin below it: coordinated drift is covered
instead by pinning the value and the direction, so restoring parity
trips a test rather than silently re-widening the blind spot. The
+25/channel case gives way to a two-sided bracket at +7/+8, and two
samples of a 2s indigo-over-white dismissal — a distance deliberately
between the new gate and the old — pin the regression itself.
The parity pin stopped the two constants drifting apart but not
together: the behavioral tests bracket the tolerance only loosely from
above, so raising both flow-pixels' PIXEL_THRESHOLD and screenshot-diff's
DEFAULT_THRESHOLD to 0.2 widened the per-pixel gate — the whole motion
oracle — with the suite still green. Assert the shared value alongside
the mirror.
Dropping `&& settled.converged` from either arm of the crop settle
mapper left the suite green: without it a crop snapshot whose settle
never converged reports clean on the skipped arm, and swaps the
timed-out note for an unavailable one on the pixel-dark arm — blaming
a missing backend for a screen that simply never stopped moving. Both
shapes are now driven through the real settle and frame resolution — a
churning hierarchy that never matches two fingerprints, and a dark
capture whose sticky verdict survives a restart — with the full reason
strings, read counts, and capture counts asserted, so each mutation
fails exactly its own test.
The settle-timeout catch's timed-out half — the only report that a
snapshot was captured while the hierarchy source never answered — was
pinned by nothing: collapsing the catch to settled left the suite
green while an update-baselines run adopted a screen never proven
still under a bare, healthy-looking write. The new case hangs every
hierarchy read, drives the real settle to its typed timeout at exactly
the action deadline, and pins the full degraded reason, the elapsed
budget, the untouched capture backend, and the written file — leaving
both halves of the ternary independently load-bearing.
The latch's timed-out override and its outage catch were pinned, but
the in-loop guard covering the remaining pixel-dark shortfalls —
unavailable, and skipped without convergence — was held by nothing:
widening it to every verdict left the suite green while a snapshot
whose stillness was already proven picked up a spurious unavailable
note. The new case drives the real machinery through a stale-but-
settled first round and four completed pixel-dark freshness retries,
asserting the exact undegraded write plus the 15-read/6-capture shape;
the mutant fails on the suffix and the counts alike.
The revalidation read is the combined settle's newest await point and
its abort exit had no direct coverage — every abort case landed on a
different read. Fabricating a converged-and-fresh result there turned a
cancelled run into a dispatched gesture recorded as a pass, with the
raw-coordinate shape uniquely exposed: it accepts settled pixels
without consulting freshness. The new case reaches a genuine settled
verdict on a capture-capable device, aborts exactly on read 3, and
pins the skip status, the run-aborted reason, no dispatched gesture,
and the capture and read counts.
The raw-coordinate path's abort guard is the only thing turning a
cancelled settle into the uniform aborted skip, and it had no abort
coverage: replacing it with a literal-point return dispatched the
gesture to the device after the client gave up, recorded as a pass —
runTap re-checks nothing before invokeOnDevice. The new case aborts
inside the settle's pair-completing read and pins the skip status, the
run-aborted reason, and that no gesture-tap reaches the device.
Deleting the visual === 'settled' disjunct left the suite green: the
tests named for it asserted only ok and coordinates, both satisfied by
the terminal best-effort dispatch at deadline exhaustion. The existing
pair now pins the dispatch instant (the first settled round at the 5s
combined phase window, not the 7.5s deadline), and a new pair covers
the transition the clause exists for — a source that stops hanging and
starts failing after pixels already settled taps the literal point it
never needed the tree for, pinned by ok, dispatch time, and a read
count proving the failing source is never consulted.
settleSnapshot keys its stale-settled override on the exact string the
pixel loop emits at budget exhaustion, but every snapshot test scripted
settleTree wholesale — relabelling the sleep-clamp arm to unavailable
left the suite green while a stale-settled update-baselines run wrote a
bare baseline with the degradation silently dropped. The new test
drives the real settleTree and real settleSnapshot end to end: a
stale-but-settled first round, a freshness retry whose animator burns
the pixel budget through the sleep-clamp arm, and the degraded write
asserted as the full reason string, pinned by read and capture counts.
Every comparison test fed the motion oracle grey — each channel term
could be deleted independently, and misindexing blue onto alpha (the
byte the docstring says is ignored) survived too, admitting motion
confined to one channel: a navy scrim fading in over black. Full-frame
red-only, green-only, and blue-only changes (plus the mid-amplitude
navy case) now register as motion, and an alpha-only change registers
as still — killing each channel-drop mutant and catching the alpha
misindex from both directions.
An already-spent caller deadline collapses into timed-out where the
flow-actions twin splits out not-attempted — deliberate here, but
undocumented and unpinned: relabelling the arm to aborted left the
suite green while turning an outage-fallback snapshot from a degraded
pass into a skip blaming a cancellation that never happened.
The collapse is now justified in prose (the outcome feeds only snapshot
degradation reporting, with no freshness contract for the twin's split
to protect, and the note text is honest for both expiry shapes) and
pinned twice: a zero-budget settlePixels call resolves timed-out with
no backend consulted, and the real fallback driven through runSnapshot
after a window-consuming outage settle degrades instead of skipping.
The simulator-server writes its temp PNG before replying, and the reply
is the only place the path is learned — so an abort landing mid-capture
severed the fetch, capturePixels soft-failed without ever learning the
path, and the file was orphaned, breaking the stated ownership contract
(the capture promise decodes and deletes its own temp file when it
eventually completes).
captureFile now passes no signal, matching the Chromium arm that never
took one: cancellation stays responsive because settleWithin abandons
the promise, while the capture self-completes on its own bounds, learns
the path, and cleans up in the finally on every path.
The pixel phase's not-attempted arm hardcoded treeFresh: true on the
grounds that zero time passed since the converged pair — false whenever
the awaited capability probe itself spent the budget: a cold iOS
runtime-kind lookup is a real simctl round trip, so the pair could be
~7s old while the settle reported it safe, and the verdict flipped from
honest to dishonest on whether that probe resolved just before or just
after the deadline.
The arm now compares the pair's convergence instant against the pixel
window: a window already spent at convergence means the probe answered
from cache and staleness is bounded by the final-read reserve, while a
window the probe consumed marks the tree unsafe — the same reasoning as
the probe-deadline sibling.
A tree-phase read outliving the source read budget returned the carried
treeFresh: after one successful read, a hung second read handed back a
tree up to the full 7.5s budget old while reporting it safe for selector
coordinates. scroll-to — the one deadline-less settleTree consumer — was
the visible casualty: its stale-round skip never fired for the case it
was written for, so one 8s hierarchy read dispatched a swipe off the
stale tree and misread the scroll as ended.
The hung-read exit now returns treeFresh: false. That is unconditional
because no path reaches it with a recent successful read in hand: every
phase window is capped by the read budget, so a non-first poll only
launches with budget open, and a first read starting past the budget
holds no successful read and throws instead. The treeFresh docstring
gains the fourth staleness source.
One combined settle on a screen with perpetual tree-invisible motion
polled captures to the caller's hard deadline minus the 250ms reserve,
consuming the entire action budget: waitForFrameResult got a single
resolution attempt instead of retrying across the window, a late retry
whose no-read window closed threw FlowTreeSettleTimeoutError out of the
step, and every directive on such a screen paid ~6s of added wall clock.
Polling for a pixel pair now stops at the ordinary phase window, the
same boundary tree polling already honors; the hard ceiling bounds only
work in flight when it closes — a first capture's first-frame-aware
budget plus the one warm capture completing its pair. After a
phase-bounded pixel timeout the mandatory revalidation read runs on the
ordinary source read budget instead of being cut to the reserve, and
waitForFrameResult treats a late retry's settle timeout as deadline
exhaustion (best-effort resolve from the last settled tree) rather than
a step error, matching the throw taxonomy.
Collapsing COMBINED_HARD_TIMEOUT_MS to the 3s tree window left the
suite green while cutting the slow-first-frame capture the constant's
pixel term exists for — scroll-to's round-0 settle, the one combined
settle with no caller deadline, silently degraded instead of proving
stillness. Drive a 6.25s first capture plus a matching warm capture
through the deadline-less settle and assert full convergence, and pin
the constant's composition directly.
The crop mapper's skipped arm was held by nothing: deleting it sent a
converged tree-only settle to the timed-out fallback, stamping every
Vega and Apple TV cropOn snapshot with a degradation note for a pixel
phase that is architecturally absent. Drive runSnapshot through the
mapper with the no-backend settle shape and assert the exact undegraded
baseline-write reason.
Collapsing the finalTreeDeadline ternary to the 250ms reserve left the
suite green while cutting every ordinary-latency post-pixel read short:
a 400ms uiautomator revalidation after a fast pixel phase came back
non-fresh, the verdict selector resolution refuses, wasting a round per
settle. Drive the everyday scenario and assert full convergence on
exactly three reads and two captures.
The two pixelsDiffer colour tests probed only the extremes, leaving the
band where a real transition lives — a dim, a fade, a crossfade —
unexercised: a 5x threshold mutant swallowed a full-frame 120-per-channel
change with the suite green. Bracket the tolerance from both sides
(grey120 registers, +25 stays still) and assert the documented mirror of
screenshot-diff's DEFAULT_THRESHOLD directly, exporting both constants.
Every existing settlePixels test fed a constant frame, hung the capture,
or aborted, so nothing past the first comparison was pinned — deleting
the predecessor advance left the suite green while a transition coming
to rest looking different from its first frame would burn the whole
window and report timed-out. Drive A/B/C/C through the real settle and
assert settled on exactly four captures.
scroll-to is the only settleTree caller that supplies no action
deadline, so its tree reads fell back to the 5s combined-phase cap —
below the budgets the sources themselves get (getHierarchy 15s,
uiautomator up to 20s) — and a healthy 5.2s hierarchy read failed only
scroll-to steps as a fabricated settle timeout. The unowned fallback now
matches DEFAULT_ACTION_TIMEOUT_MS, putting the timeout cliff at the same
7.5s every deadline-owning caller already has.
settleSnapshot's stale-settled latch treated every later reading as
unable to take back established stillness, so a retry that re-ran the
whole pixel phase without re-proving a matching pair was discarded and
the deadline exit wrote an undegraded baseline — one transient slow
post-pixel read made a mid-animation capture look clean under
--update-baselines. A completed retry ending timed-out now returns that
degradation over the latch (the capture happens after the settle, so the
retry describes the captured screen better than the stale claim); only
pixel-dark retries — outages and hard tree timeouts — still honor it.
When the tree pair converges inside the final-read reserve of the
caller's deadline, the first capture was never launched — no backend was
consulted and zero time passed — yet the settle reported visual
"timed-out", the verdict snapshots degrade on. Align the code with the
SettleResult contract: a phase that never ran returns the sticky visual
verdict (skipped by default) with the still-fresh converged tree, and no
revalidation read is spent. Budget exhaustion after a capture ran stays
an honest pixel timeout.
A combined settle whose healthy tree read consumed the caller's whole
budget threw an untyped Error from the capability-probe deadline check —
outside the settle error taxonomy, hard-stopping steps whose tree had
just converged. The budget check now distinguishes a probe that was
never consulted (no time passed — the converged tree is returned fresh)
from a probe that was awaited and outlived the wait (returned non-fresh),
so raw-coordinate gestures dispatch best-effort and snapshots degrade
honestly instead of erroring.
The existing rejection tests fail every read (an outage, thrown before
the post-pixel stage) and the restart tests move the tree via a
successful differing read, so a describe blip landing exactly on the
revalidation read exercised an untested arm: re-seed with the stable
fingerprint and restart. Two tests now pin it by read/capture counts —
the seeded restart converges on one extra read (5 total), the pixel
phase re-runs after the restart (4 captures), the blip surfaces nowhere,
and the settle returns fully converged and fresh.
The post-pixel tree revalidation gets as little as a 250ms reserve, which
one slow uiautomator/full-hierarchy read can exceed on a healthy source,
and review showed settle outcomes leaking into step failures. Settling
now stabilizes best-effort and never fails a step by itself; the only
settle-caused failure left is a genuine tree-source outage (every read
in the window failed):
- Snapshots retry a settled-but-stale settle for freshness on the shared
action deadline (the planned cropOn selector will consume the tree),
accept the stale-but-settled screen at exhaustion, and no longer
refuse --update-baselines on a timed-out settle — the write proceeds
with a degradation note instead of failing.
- Raw-coordinate tap/long-press accept a fresh tree OR settled pixels,
retry to the action deadline, then dispatch anyway: literal
coordinates depend on nothing the settle failed to prove.
- Selector resolution prefers fresh trees while budget remains and falls
back to the last valid settled tree at exhaustion, so a slow freshness
read cannot masquerade as "element not found".
- scroll-to skips a stale round instead of failing the step.
So callers can trust visual:"settled", settleTree downgrades it to
"skipped" on a tree-phase restart: the tree moving after the pixels
matched proves those pixels stale.
Fixes#644.
## Before
```
$ argent flow run echo-only # .argent/flows/echo-only.yaml: two `echo` steps
[Tool:flow-execute] No booted device found. Pass a device id or platform explicitly.
Available devices: 18BE573F-… (ios, Shutdown), <70 more>
```
and when a device *was* booted:
```
PASS on 18BE573F-… — 0 passed, 0 failed, 0 errored, 0 skipped
```
## After
```
$ argent flow run echo-only # nothing booted
Flow "echo-only"
› first step
› second step
PASS — 0 passed, 0 failed, 0 errored, 0 skipped (no test steps)
```
## Symptom 1 — the device
Whether a device is needed is decided from the flow's own steps, before
resolution.
Classification is **per step kind and defaults to needing one**, so a
kind added later inherits
today's behaviour instead of silently running against no device — and
the `never` binding makes
leaving a new kind unclassified a compile error, backed by a test keyed
on the step union so it
fails to build rather than to run.
Two classifications worth calling out:
- **`when` always needs a device**, whatever its body contains, because
the guard reads one itself
(its platform, or its view tree). A `when: {platform: ios}` block
containing only narration still
resolves.
- **`run` needs one without the fragment being read here.** The named
flow is resolved at run time;
resolving it a second time would duplicate that lookup and could
disagree with it if the file
changed in between. A deliberate over-approximation: composing a
narration-only fragment still
resolves a device. Stated in the code and in the tool description.
For `tool:` steps the tool's own schema decides, via a superset of the
keys the runner injects —
`device` is included because a nested `flow-execute` step drives a
device without being handed the
run's own. A tool that declares no input at all
(`stop-all-simulator-servers`) is device-free, and
reading its absent schema must not throw — it previously would have.
`ExecState.device` is nullable, which is the safety mechanism rather
than just typing: the compiler
enumerates every site that acts on a device, each of which now says so
via one `deviceEnv()`
narrowing. `flow-actions.ts`, `flow-visual.ts` and `ActionEnv` are
untouched. The executor also
re-checks each step against the **same predicate**, so if the scan and
the executor ever disagree
the step reports it instead of failing obscurely deeper in.
A run that resolved no device reports `device: ""`, and neither renderer
claims it ran somewhere.
## Symptom 2 — the counters
Reproducing this showed the issue's framing is slightly off: `--json`
doesn't only disagree with the
text, it disagrees with **itself** — counters all zero while its own
`steps[]` marks both steps
`"pass"`. So the CLI text was a faithful rendering of the server's
counters.
**The counters are right and stay as they are.** Narration is
deliberately excluded, and it should
be: after a hard stop every remaining step *including narration* is
reported skipped, so counting it
would inflate `skipped` on every real failing run — a worse and far more
common lie than the vacuous
one being reported here. It would also contradict the JUnit model in
#578, where echo is
`<system-out>` and not a `<testcase>`.
What was missing was saying *why* zero. Both the CLI and the MCP
renderer now append
`(no test steps)`, and only on a pass — a cancelled run can be a FAIL
with all four counters zero,
where "no test steps" would read as though the failure had no cause.
No wire format changes: `steps[].kind === "echo"` already distinguishes
counted from uncounted steps,
so the JSON is self-describing without widening the status enum.
## Conflicts, for whoever lands second
This is a hot area — seven open PRs touch `flow-run.ts`.
- **#578** rewrites the CLI summary line I also change (one hunk, one
7-line function, semantically
independent — resolution is keep-both), and its
`attachFailureDiagnostics` dereferences
`env.device.platform`. **It will need a null-device guard**, or every
failure in a device-less flow
silently degrades to `read-failed` diagnostics. Happy to rebase onto it
rather than the reverse.
- **#585** rewrites `resolveRunDevice` at the same seam and edits the
same description sentence.
- **#538** rewrites `flow-device.ts`. Worth noting it makes
`resolveFlowDevice` reject a physical
iPhone for flows — skipping resolution means a narration-only flow now
passes on a
physical-iPhone-only host, which I think is desirable but is a second
behaviour change on this seam.
All new tests are in new files, since every existing flow test file in
the blast radius is rewritten
by an open PR.
## Behaviour changes a reviewer should weigh
- A device-free flow no longer prints `PASS on <id>` **even when a
device is booted**. That is the
point: otherwise an identical flow reports differently in CI (no
simulator) than on a laptop with
one open. An explicit `--device` is still honoured and still named.
- `--platform` is ignored for a device-free flow, where it previously
errored with nothing booted.
- A 0.18.1 CLI against a newer server would print `Flow "x" on ` —
cosmetic version skew.
## Verification
Live against the built CLI: device-less flow with nothing booted (the
reported bug) and with a device
booted (identical output, no ambient attribution); `--json` showing
`device: ""`; explicit `--device`
still naming it; a mixed flow and a composed flow both still demanding a
device; a mixed flow with a
device booted still reporting `PASS on <id> — 1 passed`.
24 new tests. tool-server 3103 passed, CLI 284 passed, MCP 78 passed —
no existing test modified.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
> **#697 is stacked on this branch** — it conflicts with this one in
`flow-run.ts`'s import block, so it was rebased on top. Merge this
first.
## What
Adds the CSS combinators to flow selectors, read off frames. Any action
or condition can now scope its matches by where an element sits relative
to another:
```yaml
- tap: { text: Delete, within: { id: profile-card } } # CSS "A B" — inside that container
- tap: { role: Switch, next: { text: Wi-Fi } } # CSS "A + B" — the nearest follower
- assert: { visible: { role: Button, after: { text: Danger zone } } } # CSS "A ~ B" — any follower
- assert: { hidden: { any: true, within: { id: empty-state } } } # CSS "*" — the universal selector
```
| Scope | CSS | Reads as |
| --- | --- | --- |
| `within: <sel>` | `A B` | the match's frame sits inside the scope
element's frame |
| `after: <sel>` | `A ~ B` | the match follows the scope element in
reading order |
| `next: <sel>` | `A + B` | as `after`, narrowed to the nearest follower
of each anchor |
Plus `any: true` — the CSS `*` universal selector, accepted only
alongside a scope and never beside `text`/`id`/`role`.
The child combinator `>` has deliberately **no analog**: flow trees
flatten to leaves under one synthetic root, so parent/child structure
does not survive to replay.
## How
- **Geometric, not ancestry-based.** Frames are what survives
flattening, so every scope is resolved from them by `findAll` in
`ui-tree-match`. The synthetic root never counts as a scope element, and
every relation requires a *distinct* node (nothing scopes itself).
- **Reading order is row-band aware.** An element follows the anchor
when it starts below the anchor's bottom edge, *or* shares its row band
and sits entirely to its right. That band rule is what makes `{ role:
Switch, next: { text: Wi-Fi } }` resolve the Wi-Fi row's own switch even
though the taller switch's frame starts a couple of pixels *higher* than
the label's — a raw top-y comparison gets that backwards. `followKind`
is the single classification both the follower test and the nearest-pick
grouping read, so they cannot disagree.
- **`next` is the union over anchors**, exactly as CSS `A + B` is: three
rows on screen give three switches, one per label.
- **Scopes compose and nest**, capped at `MAX_SELECTOR_SCOPES` (6)
counted across the whole relation *tree* — a size bound, not a depth
bound, because three relations branch and the runner's loose-alternative
expansion is exponential in the bare-string scopes it finds.
- **Flow-YAML only.** `await-ui-element`'s strict schema rejects
`within`/`after`/`next`/`any`, so a scope can never be silently dropped
on that path.
- Adding a fourth relation is a **compile error**: `RELATION_RESOLVERS`
is a `Record` keyed by the relation union, so a new scope cannot be
wired through parse/serialize/reporting and then ignored by the match
engine.
## Testing
- **Unit:** full tool-server suite green — **3006 passed / 1 skipped**.
New coverage in `ui-tree-match.test.ts`, `flow-utils.test.ts`,
`flow-selector-scopes.test.ts` (renamed from
`flow-within-selector.test.ts`, now covering all scopes) and a new
`flow-skill-docs.test.ts` that parses every YAML snippet in the skill so
the docs cannot rot. Each fix is mutation-checked: reverting it fails at
least one test.
- **Real-device E2E** (Pixel 3a, Bluesky on Metro 8081, via a branch
tool-server): three flows, 20 steps, all green — geometric ordering both
ways, two `text` asserts that pin `next`'s pick *by identity* out of
five candidates, empty-scope `hidden` semantics, and two observable taps
(`next`-scoped and `any: true`-scoped) that navigated to the *adjacent*
topic rather than the anchor's own.
- **iOS not covered.** Native-devtools does not attach on either booted
simulator in this environment, so *every* flow selector fails there — an
unscoped `{ text: "Video" }` fails identically. Verified environmental,
not caused by this branch: the globally-installed argent reports the
same, direct `SIMCTL_CHILD_DYLD_INSERT_LIBRARIES` injection also fails
to connect, and it survives a forced simulator reboot on both iOS 18.3
and 26.5. Worth re-running the iOS leg once that is healthy.
## Notes for reviewers
Two deliberate deviations from CSS, both documented in the skill:
- **`next` is looser than `+`.** Where `A + B` matches nothing unless
the immediate sibling is a `B`, `next` keeps looking and returns the
nearest match further on. That is what makes it survive the wrapper and
spacer nodes a flattened tree is full of — but a row *missing* the
control you asked for resolves to the next row's rather than failing.
Happy to switch to strict `+` semantics if the loud failure is
preferred.
- **"follows" is not transitive.** Against a tall anchor an element can
follow something that follows the anchor without following it directly,
so nesting `after` scopes is not the same as chaining `~`.
Other things worth a look:
- `nearestAfter` and `afterTester` are direct scans, not indexed. An
earlier sorted index was a *second* implementation of the nearest rule
and the two disagreed on frames near the tolerance; it also measured
2-13x slower than the scan on realistic shapes, with a prune that
provably never fired. The quadratic tail is documented with measured
numbers: 1.2 ms for a single-anchor `next` at the 12k-node tree cap,
~555 ms only when the anchor selector matches essentially every node.
- A bare-string *anchor* cannot self-correct the way the loose fallback
usually does — a decoy `testID` matching your anchor text wins the
identifier pass whenever anything sits after it. The skill tells authors
to spell sibling anchors as maps.
- `MAX_SELECTOR_SCOPES = 6` replaces the previous
`MAX_SELECTOR_WITHIN_DEPTH = 10`, so a `within` chain deeper than 6 that
parsed on the earlier commits of this branch now rejects. Nothing
released depends on it.
---------
Co-authored-by: Hubert Gancarczyk <claude-hubert.gancarczyk@swmansion.com>
`snapshot` gains an optional `cropOn: <selector>` — diff one element's
region instead of the whole screen:
`- snapshot: { name: cart-total, cropOn: "Order summary" }`
- The selector resolves like a directive target: settled tree,
auto-wait, the standard
not-found failure. Selector-only — no point form (a point has no extent
to crop to).
- The frame resolves **before** the capture, and the **cropped** image
is what gets
compared, stored as the baseline, and reported as the `current`
artifact. The pixel
rect is the normalized frame × capture size; the crop is a pngjs
`bitblt` written to
a swept scratch dir and registered only when a failure returns it.
- The baseline key stays on the **full** capture's WxH (device-class
identity), so the
previously unreachable `dimensionMismatch` guard becomes load-bearing:
an element that
grew or shrank fails with "nothing was compared" plus a
crop-fixed-size-containers
hint. Deliberately strict — a 1px layout drift fails on dimensions, not
content (the
same tradeoff as Playwright element screenshots).
- A tree-source outage while resolving `cropOn` errors the step instead
of degrading to
a full-screen compare against a cropped baseline.
- No tool-schema changes; `snapshotKey` and `--output` behavior ride
along unchanged.
Tests: parse/serialize round-trip + rejection cases, and runSnapshot
coverage with real
PNGs (crop dims, key vs. content, artifact retention/sweep, unresolved
selector, abort,
size-drift message, sub-pixel region).
# feat(flow): rotate directive
Adds a selector-targeted two-finger rotation directive to flows:
`rotate: { on?, by }`. `on` resolves like `tap` (settled tree +
auto-wait); omitted, the rotation centers on the screen. `by` is
degrees, positive = clockwise. Distinct from `tool: rotate`, which
changes device orientation — the docs and step labels call this out.
## Design
**One continuous gesture, fixed pace, no duration option.** Fingers
orbit at ~90°/300 ms; duration derives from the angle. Unlike pinch
there is no sub-gesture chaining — any angle dispatches as one sweep, so
the angular delta is exact with no settle delays. `by` is parse-bounded
to ±3000°: the largest sweep the 10 s single-gesture envelope delivers
at that pace, derived from the constants rather than picked as policy —
rejected at parse, never silently clamped. A consequence embraced here:
frame count always scales with the angle, so a speed-sensitive recording
stays a raw `tool:` step (the create-flow skill's polish list covers the
conversion).
**Fixed centroid, constant physical radius.** Drifting the centroid
would couple pan into the turn; growing the radius would couple pinch.
Both stay fixed — the trade is a smaller orbit near edges and small
targets, in exchange for a pure rotation.
**Physical circle, not a normalized-space ellipse.** A single normalized
radius on a non-square screen traces a physical ellipse, coupling an
unintended pinch into every turn. `gesture-rotate` grows optional
per-axis `radiusX`/`radiusY` (fractions of width/height with `radiusX·W
= radiusY·H`), and the flow tree adapters surface screen dimensions via
an optional `screen` field on `DescribeTreeData` so the directive can
compute the aspect. Missing dimensions fail soft to the legacy
single-radius call rather than erroring.
**Placement reuses pinch's edge model.** The initial finger axis
(horizontal or vertical) is scored against the same per-platform
system-edge guards, preferring a start parallel to the nearest violated
edge and staying inside the target's axis-matching span. Guards only
score placement — they never shrink the radius, and a violated guard is
attempted, never rejected. The only geometry failure is no positive
on-screen orbit radius.
**Abort correctness.** `gesture-rotate` now honors `ctx.signal`: each
frame checks it, lifts the synthetic fingers with a terminal Up (so a
cancelled run doesn't leave them held on the glass), then throws with
name `AbortError`; the directive maps a mid-gesture abort to the run's
aborted skip rather than a step failure.
**Explicit `inputSchema`.** The radius requirement is a zod `.refine()`
cross-field rule, which `z.toJSONSchema` drops — so the tool ships a
derived schema plus an explicit `anyOf` (`radius` or
`radiusX`+`radiusY`) to keep invalid calls rejected at the schema layer.
**Platform rejections.** Vega is remote-driven (no touch) and Chromium
has no rotate-gesture idiom at all — both fail upfront with guidance to
drive the app's rotate controls directly.
## Tests
Unit tests cover the geometry module (duration derivation, candidate
construction, aspect handling, edge-guard ranking), the directive
end-to-end against a fake device, and `gesture-rotate`'s per-axis radii
and abort path.
Adds a selector-targeted pinch-zoom directive to flows: `pinch: { on?,
scale }`. `on` resolves like `tap` (settled tree + auto-wait); omitted,
the pinch centers on the screen. `scale > 1` zooms in, `< 1` zooms out.
## Design
**Options-map only, no bare form.** `tap: "Map"` is unambiguous; `pinch:
"Map"` isn't (in or out?), so the parser requires `{ on?, scale }` and
rejects a bare selector with authoring guidance.
**Validity checks, not policy caps.** `scale` must be finite, > 0, and ≠
1 (a no-op scale is almost certainly a mistake) — but there is
deliberately no magnitude cap. An extreme scale decomposes
logarithmically into equal-ratio sub-gestures of ≤4× each, chained with
a 250 ms recognizer-reset delay (`scale: 1e6` is just 10 gestures).
**Pure geometry module.** All placement math lives in
`flow-pinch-geometry.ts` as deterministic data-in/data-out functions (no
device access), so tests drive it with synthetic centers, spans, and
edge guards. Per sub-gesture, a horizontal and a vertical candidate are
built from the axis-matching target dimension (normalized X and Y are
different units, so never `min(width, height)`), and the better one is
dispatched.
**Edge safety scores, it never rejects.** Per-platform system-edge
guards (wider on Android's back-gesture zones) demote candidates in
ranking but never fail a step: viable travel beats edge safety (a
risky-but-perceptible gesture beats a safe no-op the recognizer
ignores), then fully-safe with the most travel, then motion parallel to
the violated edge (edge swipes need motion *away* from the edge). Guards
are resolved once per directive and passed as data — the seam for a
future per-device query. The only geometry failure is literally zero
on-screen finger travel; a tiny target is still attempted.
**Centroid drift near edges.** When expanding fingers would leave the
screen, the centroid drifts inward — only on the moving axis, and only
as far as needed. `gesture-pinch` grows optional
`endCenterX`/`endCenterY` to interpolate the centroid alongside
separation; omitted, behavior is unchanged.
**Open-loop by design.** There is no "current zoom" to read back, so the
directive doesn't try to verify the multiplier — flows assert on the
resulting UI instead.
**Platform rejections.** Vega is remote-driven (no touch input) and
Chromium has no uniform pinch-zoom mapping for a desktop app (ctrl+wheel
or app-specific controls) — both fail upfront with guidance to drive the
app's zoom UI directly.
## Tests
Unit tests cover the geometry module (decomposition, candidate
construction, edge-guard ranking), the directive end-to-end against a
fake device, and the `gesture-pinch` centroid-drift interpolation.
Flow reports rendered every step flat, hiding which steps ran inside a
`when:` block or a `run:` fragment. The tool-server now stamps
`StepReport.depth` (+1 per block directive, omitted at top level) and
the CLI/MCP renderers indent step labels and under-step lines by it:
```
✓ 1 launch
✓ 2 when visible "What's new" — condition met
✓ 3 tap "Skip"
✓ 4 run dismiss [dismiss]
✓ 5 tap "OK" [dismiss]
```
Invariants:
- **Wire-compatible both ways** — a flow with no block directives
reports byte-identical to the pre-depth shape; a pre-depth server
renders flat as before.
- **Skipped ≡ entered** — skipped/errored/aborted blocks report the same
depths as entered ones, keeping reports comparable run-to-run.
- **Untrusted wire data** — renderers clamp depth (display cap, not a
producer bound): hostile values render flat or at the cap, never throw
or allocate huge strings.
Also fixes a pre-existing bug: under-step warning/artifact lines now
stay aligned past 99 steps.
Tests pin depth on every producer path (entered/skipped/errored/aborted
blocks, `run:` success/failure markers, `run:`-in-`when:`) and renderer
alignment incl. the clamp boundary.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What
A `when:` block runs a step list only when one condition holds — a UI
condition (`exists`/`visible`/`hidden`/`text`, the await/assert shapes)
or a static `platform` test. **No else**, by design: a block exists to
restore determinism (dismiss an interstitial, reconverge), never to
test two paths. Tap-if-present is a one-step block:
`when: { visible: "Got it" }` + `steps: [tap: "Got it"]`.
No per-step `optional:` — `when:` covers every directive once. The key
is rejected at parse with a pointer to `when:`, not silently dropped (a
Maestro habit that would leave a "can't fail" step hard-stopping the
flow).
## How
- The guard reuses the assert engine at the 1s grace (`timeout` is a
parse error), so a skipped block barely costs a clean run.
- Verdicts are evidence-based: no trustworthy read in the window →
indeterminate → the step **errors** ("could not evaluate" is not
"condition false"); a transient read failure after trusted reads showed
the condition false stays a clean skip.
- `platform` folds `ios-remote` into `ios` — the parser rejects that
spelling, so the fold is the only way a guard matches a remote sim.
- A skipped block reports one line per authored step, same shape for
unmet guard, hard stop, or cancellation; everything goes through
`pushReport`, so when lines stream live. Failures inside an entered
block hard-stop as usual.
- Results carry an explicit `aborted` flag; `ok` keys off
fail/error/cancel, not skip lines.
- Nesting is capped at 20, so a cyclic YAML alias is a structured parse
error, not a stack overflow.
## Testing
`vitest` 2546 pass (17 when-specific); `tsc --noEmit` (tool-server,
argent-mcp, argent-cli), eslint, prettier clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
This adds raw normalized screen coordinates as first-class targets for
both `tap` and `long-press`. Gestures can now target either an element
selector or a point, while keeping the existing selector auto-wait and
off-screen diagnostics.
Supported forms include:
```yaml
- tap: { x: 0.5, y: 0.6 }
- tap: { on: { x: 0.5, y: 0.6 }, times: 2 }
- long-press: { x: 0.5, y: 0.6 }
- long-press: { on: { x: 0.5, y: 0.6 }, duration: 1200 }
```
## What changed
- Introduced a shared gesture-target parser for selector and point
targets.
- Added raw-point execution for `long-press` and shared point resolution
between tap and long-press.
- Validate coordinates as finite normalized `0–1` screen fractions,
rejecting pixels, malformed points, mixed selector/coordinate maps, and
unknown point keys.
- Canonicalize option-bearing coordinate gestures under `on`, keeping
option keys separate from target fields.
- Preserve coordinate targets in recording summaries and flow progress
reports instead of displaying an undefined selector.
- Added serialization validation so recording cannot persist a flow that
would fail when parsed for replay.
- Updated the flow authoring skill and tool description with the new
coordinate forms.
## Behavior notes
- Raw points act immediately and do not resolve or observe the UI tree.
- Selector targets retain settling, auto-wait, and the existing
`scroll-to` hint when an element cannot be found.
- Observation directives such as `type`, `await`, `assert`, and
`scroll-to` remain selector-only.
- The mixed spelling `tap: { x, y, times }` is now intentionally
rejected; use `tap: { on: { x, y }, times }` instead.
## Validation
- Full tool-server suite: 2,618 tests passed across 266 files.
- Added parser, serialization, recording, touch execution, Chromium
execution, reporting, and malformed-input coverage.
- Tool-server build, test typecheck, ESLint, Prettier, and skill
validation passed.
## Summary
`run:` accepted **fragments only** — an e2e flow (one starting with
`launch:`)
was rejected as a `run:` target. This lifts that: an e2e flow can now be
nested
via `run:`, its `launch` running inline (a real terminate + relaunch on
native).
Composition becomes uniform — any flow can be a reusable building block.
## What changed
- **`flow-run.ts`**: dropped the `isE2eFlow` rejection in `execRunStep`;
cycle /
`MAX_RUN_DEPTH` guards still bound recursion.
- **`flow-add-step.ts`**: the recorder now captures a recorded
`flow-execute` of
an e2e sibling as a portable `run: <name>` directive (was: raw `tool:`
step).
- **`flow-utils.ts`, `argent-create-flow/SKILL.md`**: updated docs/skill
that
said e2e flows can't be `run:` targets.
## Known limitation — Chromium
Chromium `launch` boots an Electron app from a *path*, wired to the
top-level
flow only — so a nested **chromium** e2e flow doesn't boot its own app.
Native
nesting is correct; chromium nested e2e is unsupported.
## Testing
Rewrote the two affected tests (composition + recorder). Full flows
suite green
(349 tests); `tsc --noEmit` clean.
## The bug (found in the 0.15.0 full-surface QA, on both the Android
emulator and a physical S24)
Recording a `gesture-tap` on an icon-font tab-bar item wrote a flow step
that displays as an **empty selector**, with the coordinates gone:
```yaml
steps:
- tap:
text: # ← actually U+E163, an icon-font Private Use Area glyph
```
and `flow-execute` then **silently passed** against target `""`, tapping
an element no reader of the flow can predict. Root cause (hexdump of the
QA-recorded file: `ee 85 a3` = U+E163): icon fonts expose their glyph as
the element's accessibility label; `deriveSelector` accepted any label
that survived `trim()`, so a PUA code point counted as "stable text".
This violated the rewrite's own documented contract — *"otherwise
coordinates are kept with a warning"* — whose fallback path existed and
worked, but never ran for this case.
## The fix — one predicate, three layers
`hasVisibleText()` (strip Unicode `Cf` format / `Co` private-use / `Cc`
control, then trim; require something left):
1. **`deriveSelector`** — invisible-only label/value is not stable text;
fall through to role, then to the recorder's existing kept-coordinates
path.
2. **`selectorToYaml`** — the empty-text guard previously covered only
the bare-string spelling; it now also rejects empty/invisible text on
the strict `{text: …}` map form, so a hand-built selector fails loudly
instead of writing an unreadable flow file.
3. **`selectorSchema`** — parse-side refinement, so flow files already
recorded broken on 0.15.0 fail with an actionable message ("text must
contain at least one visible character … select by identifier or role
instead") instead of false-passing at replay. This also hardens the
`await-ui-element` tool surface, which shares the schema.
Text that merely *contains* an icon glyph alongside real text ("⟨U+E163⟩
Explore") stays valid throughout.
## Verification
- Unit: `deriveSelector` PUA/ZWSP/mixed cases, serialize + parse
rejection round-trips (explicit `\uXXXX` escapes, no invisible bytes in
the test source).
- Full tool-server suite: **2667 passed**.
- **Live on the Pixel_9 emulator against the original QA repro** (my-app
tab bar, same coordinates): recording now produces `tap: {x: 0.75, y:
0.947}` with the documented *"kept coordinates (brittle)"* warning — the
role fallback was correctly caught by the existing
resolves-to-a-different-element guard — replay passes 2/2 on the
coordinate targets, and executing the old broken YAML now fails parse
with the visible-character error.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
- Add `matches` as a case-sensitive JavaScript regex comparator for
`await` and `assert` text conditions.
- Support `{ text: { matches: ... } }` in every flow selector slot,
combined with `id` and `role`.
- Preserve regex matchers through parsing, YAML serialization, recording
summaries, and run reports.
## Correctness
- Prevent empty-matchable patterns from matching unlabeled nodes or
empty assertion text.
- Reject unknown raw `await-ui-element` selector keys instead of
silently dropping constraints.
- Reject selector combinations YAML cannot round-trip without changing
their semantics.
- Document native label aggregation accurately and keep action ranking
consistent.
## Maintenance
- Share selector-field and regex-syntax validation across both matcher
parse paths.
- Compile selector regexes once per tree walk and full-match regexes
once per ranking pass.
- Make comparator serialization exhaustive, extract the duplicated flow
test harness, and scope slow-test timeouts narrowly.
## Verification
- Full tool-server suite: 2,588 tests across 263 files.
- Source and test typechecks, ESLint, Prettier, and skill-copy
validation pass.
Flows can now express double-taps declaratively:
```yaml
- tap: { on: Login, times: 2 } # selector double-tap
- tap: { x: 0.5, y: 0.57, times: 2 } # coordinate double-tap
```
The selector nests under `on` so option keys never share a map with
selector fields — zod would silently strip them; mixing errors with a
hint instead. `times: 1` normalizes to absent and the serializer emits
the minimal spelling, keeping parse/serialize exact inverses.
The taps dispatch as **one gesture** via `gesture-tap`'s new
`clickCount` (1–10): N Down/Up pairs ~100 ms apart on touch, escalating
CDP `clickCount` per click on Chromium (what makes `dblclick` actually
fire) — never N separate calls whose RPC gaps could miss the OS
double-tap window.
The recorder carries a recorded `clickCount` into `times`, so a recorded
double-tap can't silently replay as a single tap.
Press-and-hold on an element — context menus, reorder activation — with
the same resolution behavior as `tap` (settled tree, auto-wait, no
auto-scroll) and the same options shape: a bare/strict selector, or
`long-press: { on: <selector>, duration: <ms> }` with the selector
nested under `on` so option keys never mix with selector fields.
The default 800ms clears both platforms' recognizers (iOS 500ms,
Android ~400ms). Touch platforms dispatch one `gesture-custom` train
(`Down`, then `Up` delayed by the duration) so the hold length is
exact; Chromium maps to a mouse press-hold-release (`gesture-drag`,
from == to) — a desktop context menu is a right-click and is
deliberately not aliased. Vega rejects upfront with the other touch
directives. No coordinate form: a point long-press stays the
`gesture-custom` escape hatch.
# Flow E2E
This PR turns flows from replayable tool-call recordings into a
declarative, portable e2e test system that runs headlessly via `argent
flow run <name>` — no LLM in the loop, non-zero exit on failure,
suitable for CI.
Agents can still replay flows via `flow-execute` and get the full output
from each step after it finishes.
## Flow format
Flows are YAML files in `.argent/flows/`, now built from **directives**
interpreted by the runner rather than raw tool calls:
- `launch` — terminate + relaunch the app under test (per-platform app
id map supported); its presence marks a flow as **e2e** (controls its
own start state) vs. a **fragment** (runs against current state,
declares an `executionPrerequisite`, composable via `run:` with
cycle/depth guards).
- `tap`, `type` (focus handshake + optional submit), `scroll-to`
(momentum-free, increments by half the container, `within:` for nested
scrollers), `await`/`assert` (`visible`/`exists`/`hidden`/`text`
conditions), `wait`, `snapshot`, `echo`, and raw `tool:` as escape hatch
— still fully supported, so previously recorded flows keep working
unchanged.
- **Selectors** are `{ text, identifier, role }`; a bare string is a
loose selector resolving identifier-first, then visible text. Any
directive failure hard-stops the flow; later steps report `skip`.
## Execution engine
- Flows store no device id — the runner binds a device at replay
(explicit `--device`/`--platform` or the single booted one).
- Selector resolution runs against **platform-unified UI trees**: iOS
native view hierarchy (full testID coverage, gated on native-devtools
connecting), Android full accessibility hierarchy (full resource-id
coverage), Chromium CDP DOM, and the Vega automation toolkit. A shared
flatten/text-hoisting pass lets `text` assertions read what an
identified container visibly shows. Vega is remote-driven, so touch
directives fail with guidance to use `tv-remote`/`keyboard`.
- Steps wait for the tree to **settle** (two consecutive identical
reads) before acting; `launch` additionally gates on platform
tree-source readiness instead of silently degrading.
## Visual snapshots
`snapshot` diffs a full-resolution screenshot against baselines in
`.argent/flows/__baselines__/<flow>/`, keyed by platform + resolution,
with configurable `maxMismatch` and `--update-baselines`. The status bar
is pinned for the whole run (iOS `simctl status_bar`, Android demo mode)
so clocks/battery don't drive diffs. Baseline/current/diff images are
returned as artifacts on failure (and the written baseline on first
capture/update); a clean pass carries none, keeping reports and agent
context lean.
## Recording
The record-live workflow (`flow-start-recording` → `flow-add-step` →
`flow-finish-recording`) now emits directives automatically: coordinate
taps are rewritten to portable selector `tap:` steps (selector captured
from the pre-tap tree), a leading `restart-app` becomes `launch:`,
`flow-execute` of a sibling fragment becomes `run:`, and device ids are
stripped. The skill instructs the agent to polish the saved YAML
afterwards (fold tap+keyboard into `type:`, swipes-to-reach into
`scroll-to:`, awaits into sugar form).
## Also included
- `argent flow run` / `argent flow list` CLI with per-step report
rendering, artifact materialization, and `--json` output.
- New structured failure codes for flow validation, device resolution,
composition, and devtools readiness.
- Extensive unit test coverage (tree adapters, selector matching,
scrolling, composition, type focus, snapshots, status bar).
## Summary
Adds a `await-ui-element` tool that blocks until a UI condition is
satisfied or a timeout elapses.
This is one of the most impactful missing tools. Without it, agents have
to poll manually with `screenshot → describe → check` loops — slow,
token-heavy, and unreliable (fixed `sleep` delays either over-wait or
fire before the screen settles). `await-ui-element` moves the poll loop
server-side: one call in, a definitive verdict out.
Works on **iOS, Android, and Chromium (CDP)**, polling the same
accessibility / DOM tree as `describe`.
## Conditions
- `visible <selector>` — wait until an element appears on screen
- `hidden <selector>` — wait until an element disappears
- `exists <selector>` — wait until an element is in the tree
- `text <selector> <expectedText>` — wait until the matched element
contains the text
**selector** is `{ text?, identifier?, role? }` — every field provided
must match (case-insensitive substring). Polls internally; default
`timeoutMs` **5000**, default `pollIntervalMs` **400**. Returns `{
success: boolean, elapsed: number }`, with a `note` explaining what was
seen on timeout.
```json
{ "udid": "<UDID>", "condition": "visible", "selector": { "text": "Continue" } }
```
Also usable as a step inside `run-sequence`, so a tap → await-ui-element
→ tap chain runs in a single call.
## Notable details
- **Multi-match aware** — a substring selector can hit several nodes, so
conditions evaluate the whole match set: `visible` holds if any match is
on-screen, `hidden` only if none is (a zero-area container can't flip
the verdict).
- **`hidden` guard** — a selector that matches nothing counts as
already-hidden; the `note` flags when the selector never matched, so a
typo doesn't read as a silent success.
- **Deadline-clamped poll** — a large `pollIntervalMs` can't overshoot
`timeoutMs`.
- **Abort-aware** — both the `time` sleep and the poll loop stop
promptly on cancellation.
## Changes
- `tools/await-ui-element/index.ts` — new tool (factory, mirrors
`describe`'s service resolution)
- `utils/setup-registry.ts` — registers the tool
- `tools/run-sequence/index.ts` — allows `await-ui-element` as a step +
arg-doc + example
- `test/await-ui-element.test.ts` — new test suite
- `skills/rules/argent.md` + `argent-device-interact/SKILL.md` —
document `await-ui-element`, replace the "don't poll `screenshot`"
guidance with it
## Test plan
- [x] `time` sleeps and succeeds without touching a device
- [x] `visible` succeeds once the element appears; times out with a
diagnostic note when it doesn't
- [x] `exists` / `hidden` / `text` happy paths; `text` timeout note
reports last-seen text
- [x] `hidden` instant success **and** the "selector never matched" flag
- [x] poll sleep clamped to the deadline (large `pollIntervalMs` can't
overshoot `timeoutMs`)
- [x] prompt cancellation aborts both the poll loop and the `time` sleep
- [x] multi-match evaluation (`findAll` / `evaluateMatches`)
- [x] schema validation (required fields per condition, `durationMs`
cap)
Jakub Piasecki