The motion oracle borrowed screenshot-diff's DEFAULT_THRESHOLD, a value
sized for a baseline held across sessions and machines and so obliged to
absorb real drift. This comparison holds two captures one poll apart
from a single live session, where a static screen reads back
byte-identical: both capture paths are lossless PNG and the downscale is
deterministic. The borrowed value put the gate at ~25.5 levels/channel,
which mattered because spatially uniform change is all-or-nothing —
every pixel moves by the same amount, so it clears the gate on all of
them or on none, and MOTION_FRACTION is never the deciding term. A
cross-fade slower than ~1.4s therefore counted zero pixels and reported
settled mid-animation, over a screen still painted and still
hit-testing; and being settled rather than timed-out, degradedReason
stayed silent while updateBaselines could adopt a mid-fade frame as
golden.
0.03 (~7.6 levels/channel) keeps a margin over the +5/channel drift the
tests treat as the noise floor and moves that cliff out ~3.3x, to ~4.5s
at full contrast. It stays a rate floor rather than a duration floor, so
the comment now names the gate as the only term bounding what is left.
This supersedes the parity pin below it: coordinated drift is covered
instead by pinning the value and the direction, so restoring parity
trips a test rather than silently re-widening the blind spot. The
+25/channel case gives way to a two-sided bracket at +7/+8, and two
samples of a 2s indigo-over-white dismissal — a distance deliberately
between the new gate and the old — pin the regression itself.
The parity pin stopped the two constants drifting apart but not
together: the behavioral tests bracket the tolerance only loosely from
above, so raising both flow-pixels' PIXEL_THRESHOLD and screenshot-diff's
DEFAULT_THRESHOLD to 0.2 widened the per-pixel gate — the whole motion
oracle — with the suite still green. Assert the shared value alongside
the mirror.
Dropping `&& settled.converged` from either arm of the crop settle
mapper left the suite green: without it a crop snapshot whose settle
never converged reports clean on the skipped arm, and swaps the
timed-out note for an unavailable one on the pixel-dark arm — blaming
a missing backend for a screen that simply never stopped moving. Both
shapes are now driven through the real settle and frame resolution — a
churning hierarchy that never matches two fingerprints, and a dark
capture whose sticky verdict survives a restart — with the full reason
strings, read counts, and capture counts asserted, so each mutation
fails exactly its own test.
The settle-timeout catch's timed-out half — the only report that a
snapshot was captured while the hierarchy source never answered — was
pinned by nothing: collapsing the catch to settled left the suite
green while an update-baselines run adopted a screen never proven
still under a bare, healthy-looking write. The new case hangs every
hierarchy read, drives the real settle to its typed timeout at exactly
the action deadline, and pins the full degraded reason, the elapsed
budget, the untouched capture backend, and the written file — leaving
both halves of the ternary independently load-bearing.
The latch's timed-out override and its outage catch were pinned, but
the in-loop guard covering the remaining pixel-dark shortfalls —
unavailable, and skipped without convergence — was held by nothing:
widening it to every verdict left the suite green while a snapshot
whose stillness was already proven picked up a spurious unavailable
note. The new case drives the real machinery through a stale-but-
settled first round and four completed pixel-dark freshness retries,
asserting the exact undegraded write plus the 15-read/6-capture shape;
the mutant fails on the suffix and the counts alike.
The revalidation read is the combined settle's newest await point and
its abort exit had no direct coverage — every abort case landed on a
different read. Fabricating a converged-and-fresh result there turned a
cancelled run into a dispatched gesture recorded as a pass, with the
raw-coordinate shape uniquely exposed: it accepts settled pixels
without consulting freshness. The new case reaches a genuine settled
verdict on a capture-capable device, aborts exactly on read 3, and
pins the skip status, the run-aborted reason, no dispatched gesture,
and the capture and read counts.
The raw-coordinate path's abort guard is the only thing turning a
cancelled settle into the uniform aborted skip, and it had no abort
coverage: replacing it with a literal-point return dispatched the
gesture to the device after the client gave up, recorded as a pass —
runTap re-checks nothing before invokeOnDevice. The new case aborts
inside the settle's pair-completing read and pins the skip status, the
run-aborted reason, and that no gesture-tap reaches the device.
Deleting the visual === 'settled' disjunct left the suite green: the
tests named for it asserted only ok and coordinates, both satisfied by
the terminal best-effort dispatch at deadline exhaustion. The existing
pair now pins the dispatch instant (the first settled round at the 5s
combined phase window, not the 7.5s deadline), and a new pair covers
the transition the clause exists for — a source that stops hanging and
starts failing after pixels already settled taps the literal point it
never needed the tree for, pinned by ok, dispatch time, and a read
count proving the failing source is never consulted.
settleSnapshot keys its stale-settled override on the exact string the
pixel loop emits at budget exhaustion, but every snapshot test scripted
settleTree wholesale — relabelling the sleep-clamp arm to unavailable
left the suite green while a stale-settled update-baselines run wrote a
bare baseline with the degradation silently dropped. The new test
drives the real settleTree and real settleSnapshot end to end: a
stale-but-settled first round, a freshness retry whose animator burns
the pixel budget through the sleep-clamp arm, and the degraded write
asserted as the full reason string, pinned by read and capture counts.
Every comparison test fed the motion oracle grey — each channel term
could be deleted independently, and misindexing blue onto alpha (the
byte the docstring says is ignored) survived too, admitting motion
confined to one channel: a navy scrim fading in over black. Full-frame
red-only, green-only, and blue-only changes (plus the mid-amplitude
navy case) now register as motion, and an alpha-only change registers
as still — killing each channel-drop mutant and catching the alpha
misindex from both directions.
An already-spent caller deadline collapses into timed-out where the
flow-actions twin splits out not-attempted — deliberate here, but
undocumented and unpinned: relabelling the arm to aborted left the
suite green while turning an outage-fallback snapshot from a degraded
pass into a skip blaming a cancellation that never happened.
The collapse is now justified in prose (the outcome feeds only snapshot
degradation reporting, with no freshness contract for the twin's split
to protect, and the note text is honest for both expiry shapes) and
pinned twice: a zero-budget settlePixels call resolves timed-out with
no backend consulted, and the real fallback driven through runSnapshot
after a window-consuming outage settle degrades instead of skipping.
The simulator-server writes its temp PNG before replying, and the reply
is the only place the path is learned — so an abort landing mid-capture
severed the fetch, capturePixels soft-failed without ever learning the
path, and the file was orphaned, breaking the stated ownership contract
(the capture promise decodes and deletes its own temp file when it
eventually completes).
captureFile now passes no signal, matching the Chromium arm that never
took one: cancellation stays responsive because settleWithin abandons
the promise, while the capture self-completes on its own bounds, learns
the path, and cleans up in the finally on every path.
The pixel phase's not-attempted arm hardcoded treeFresh: true on the
grounds that zero time passed since the converged pair — false whenever
the awaited capability probe itself spent the budget: a cold iOS
runtime-kind lookup is a real simctl round trip, so the pair could be
~7s old while the settle reported it safe, and the verdict flipped from
honest to dishonest on whether that probe resolved just before or just
after the deadline.
The arm now compares the pair's convergence instant against the pixel
window: a window already spent at convergence means the probe answered
from cache and staleness is bounded by the final-read reserve, while a
window the probe consumed marks the tree unsafe — the same reasoning as
the probe-deadline sibling.
A tree-phase read outliving the source read budget returned the carried
treeFresh: after one successful read, a hung second read handed back a
tree up to the full 7.5s budget old while reporting it safe for selector
coordinates. scroll-to — the one deadline-less settleTree consumer — was
the visible casualty: its stale-round skip never fired for the case it
was written for, so one 8s hierarchy read dispatched a swipe off the
stale tree and misread the scroll as ended.
The hung-read exit now returns treeFresh: false. That is unconditional
because no path reaches it with a recent successful read in hand: every
phase window is capped by the read budget, so a non-first poll only
launches with budget open, and a first read starting past the budget
holds no successful read and throws instead. The treeFresh docstring
gains the fourth staleness source.
One combined settle on a screen with perpetual tree-invisible motion
polled captures to the caller's hard deadline minus the 250ms reserve,
consuming the entire action budget: waitForFrameResult got a single
resolution attempt instead of retrying across the window, a late retry
whose no-read window closed threw FlowTreeSettleTimeoutError out of the
step, and every directive on such a screen paid ~6s of added wall clock.
Polling for a pixel pair now stops at the ordinary phase window, the
same boundary tree polling already honors; the hard ceiling bounds only
work in flight when it closes — a first capture's first-frame-aware
budget plus the one warm capture completing its pair. After a
phase-bounded pixel timeout the mandatory revalidation read runs on the
ordinary source read budget instead of being cut to the reserve, and
waitForFrameResult treats a late retry's settle timeout as deadline
exhaustion (best-effort resolve from the last settled tree) rather than
a step error, matching the throw taxonomy.
Collapsing COMBINED_HARD_TIMEOUT_MS to the 3s tree window left the
suite green while cutting the slow-first-frame capture the constant's
pixel term exists for — scroll-to's round-0 settle, the one combined
settle with no caller deadline, silently degraded instead of proving
stillness. Drive a 6.25s first capture plus a matching warm capture
through the deadline-less settle and assert full convergence, and pin
the constant's composition directly.
The crop mapper's skipped arm was held by nothing: deleting it sent a
converged tree-only settle to the timed-out fallback, stamping every
Vega and Apple TV cropOn snapshot with a degradation note for a pixel
phase that is architecturally absent. Drive runSnapshot through the
mapper with the no-backend settle shape and assert the exact undegraded
baseline-write reason.
Collapsing the finalTreeDeadline ternary to the 250ms reserve left the
suite green while cutting every ordinary-latency post-pixel read short:
a 400ms uiautomator revalidation after a fast pixel phase came back
non-fresh, the verdict selector resolution refuses, wasting a round per
settle. Drive the everyday scenario and assert full convergence on
exactly three reads and two captures.
The two pixelsDiffer colour tests probed only the extremes, leaving the
band where a real transition lives — a dim, a fade, a crossfade —
unexercised: a 5x threshold mutant swallowed a full-frame 120-per-channel
change with the suite green. Bracket the tolerance from both sides
(grey120 registers, +25 stays still) and assert the documented mirror of
screenshot-diff's DEFAULT_THRESHOLD directly, exporting both constants.
Every existing settlePixels test fed a constant frame, hung the capture,
or aborted, so nothing past the first comparison was pinned — deleting
the predecessor advance left the suite green while a transition coming
to rest looking different from its first frame would burn the whole
window and report timed-out. Drive A/B/C/C through the real settle and
assert settled on exactly four captures.
scroll-to is the only settleTree caller that supplies no action
deadline, so its tree reads fell back to the 5s combined-phase cap —
below the budgets the sources themselves get (getHierarchy 15s,
uiautomator up to 20s) — and a healthy 5.2s hierarchy read failed only
scroll-to steps as a fabricated settle timeout. The unowned fallback now
matches DEFAULT_ACTION_TIMEOUT_MS, putting the timeout cliff at the same
7.5s every deadline-owning caller already has.
settleSnapshot's stale-settled latch treated every later reading as
unable to take back established stillness, so a retry that re-ran the
whole pixel phase without re-proving a matching pair was discarded and
the deadline exit wrote an undegraded baseline — one transient slow
post-pixel read made a mid-animation capture look clean under
--update-baselines. A completed retry ending timed-out now returns that
degradation over the latch (the capture happens after the settle, so the
retry describes the captured screen better than the stale claim); only
pixel-dark retries — outages and hard tree timeouts — still honor it.
When the tree pair converges inside the final-read reserve of the
caller's deadline, the first capture was never launched — no backend was
consulted and zero time passed — yet the settle reported visual
"timed-out", the verdict snapshots degrade on. Align the code with the
SettleResult contract: a phase that never ran returns the sticky visual
verdict (skipped by default) with the still-fresh converged tree, and no
revalidation read is spent. Budget exhaustion after a capture ran stays
an honest pixel timeout.
A combined settle whose healthy tree read consumed the caller's whole
budget threw an untyped Error from the capability-probe deadline check —
outside the settle error taxonomy, hard-stopping steps whose tree had
just converged. The budget check now distinguishes a probe that was
never consulted (no time passed — the converged tree is returned fresh)
from a probe that was awaited and outlived the wait (returned non-fresh),
so raw-coordinate gestures dispatch best-effort and snapshots degrade
honestly instead of erroring.
The existing rejection tests fail every read (an outage, thrown before
the post-pixel stage) and the restart tests move the tree via a
successful differing read, so a describe blip landing exactly on the
revalidation read exercised an untested arm: re-seed with the stable
fingerprint and restart. Two tests now pin it by read/capture counts —
the seeded restart converges on one extra read (5 total), the pixel
phase re-runs after the restart (4 captures), the blip surfaces nowhere,
and the settle returns fully converged and fresh.
The post-pixel tree revalidation gets as little as a 250ms reserve, which
one slow uiautomator/full-hierarchy read can exceed on a healthy source,
and review showed settle outcomes leaking into step failures. Settling
now stabilizes best-effort and never fails a step by itself; the only
settle-caused failure left is a genuine tree-source outage (every read
in the window failed):
- Snapshots retry a settled-but-stale settle for freshness on the shared
action deadline (the planned cropOn selector will consume the tree),
accept the stale-but-settled screen at exhaustion, and no longer
refuse --update-baselines on a timed-out settle — the write proceeds
with a degradation note instead of failing.
- Raw-coordinate tap/long-press accept a fresh tree OR settled pixels,
retry to the action deadline, then dispatch anyway: literal
coordinates depend on nothing the settle failed to prove.
- Selector resolution prefers fresh trees while budget remains and falls
back to the last valid settled tree at exhaustion, so a slow freshness
read cannot masquerade as "element not found".
- scroll-to skips a stale round instead of failing the step.
So callers can trust visual:"settled", settleTree downgrades it to
"skipped" on a tree-phase restart: the tree moving after the pixels
matched proves those pixels stale.
Fixes#644.
## Before
```
$ argent flow run echo-only # .argent/flows/echo-only.yaml: two `echo` steps
[Tool:flow-execute] No booted device found. Pass a device id or platform explicitly.
Available devices: 18BE573F-… (ios, Shutdown), <70 more>
```
and when a device *was* booted:
```
PASS on 18BE573F-… — 0 passed, 0 failed, 0 errored, 0 skipped
```
## After
```
$ argent flow run echo-only # nothing booted
Flow "echo-only"
› first step
› second step
PASS — 0 passed, 0 failed, 0 errored, 0 skipped (no test steps)
```
## Symptom 1 — the device
Whether a device is needed is decided from the flow's own steps, before
resolution.
Classification is **per step kind and defaults to needing one**, so a
kind added later inherits
today's behaviour instead of silently running against no device — and
the `never` binding makes
leaving a new kind unclassified a compile error, backed by a test keyed
on the step union so it
fails to build rather than to run.
Two classifications worth calling out:
- **`when` always needs a device**, whatever its body contains, because
the guard reads one itself
(its platform, or its view tree). A `when: {platform: ios}` block
containing only narration still
resolves.
- **`run` needs one without the fragment being read here.** The named
flow is resolved at run time;
resolving it a second time would duplicate that lookup and could
disagree with it if the file
changed in between. A deliberate over-approximation: composing a
narration-only fragment still
resolves a device. Stated in the code and in the tool description.
For `tool:` steps the tool's own schema decides, via a superset of the
keys the runner injects —
`device` is included because a nested `flow-execute` step drives a
device without being handed the
run's own. A tool that declares no input at all
(`stop-all-simulator-servers`) is device-free, and
reading its absent schema must not throw — it previously would have.
`ExecState.device` is nullable, which is the safety mechanism rather
than just typing: the compiler
enumerates every site that acts on a device, each of which now says so
via one `deviceEnv()`
narrowing. `flow-actions.ts`, `flow-visual.ts` and `ActionEnv` are
untouched. The executor also
re-checks each step against the **same predicate**, so if the scan and
the executor ever disagree
the step reports it instead of failing obscurely deeper in.
A run that resolved no device reports `device: ""`, and neither renderer
claims it ran somewhere.
## Symptom 2 — the counters
Reproducing this showed the issue's framing is slightly off: `--json`
doesn't only disagree with the
text, it disagrees with **itself** — counters all zero while its own
`steps[]` marks both steps
`"pass"`. So the CLI text was a faithful rendering of the server's
counters.
**The counters are right and stay as they are.** Narration is
deliberately excluded, and it should
be: after a hard stop every remaining step *including narration* is
reported skipped, so counting it
would inflate `skipped` on every real failing run — a worse and far more
common lie than the vacuous
one being reported here. It would also contradict the JUnit model in
#578, where echo is
`<system-out>` and not a `<testcase>`.
What was missing was saying *why* zero. Both the CLI and the MCP
renderer now append
`(no test steps)`, and only on a pass — a cancelled run can be a FAIL
with all four counters zero,
where "no test steps" would read as though the failure had no cause.
No wire format changes: `steps[].kind === "echo"` already distinguishes
counted from uncounted steps,
so the JSON is self-describing without widening the status enum.
## Conflicts, for whoever lands second
This is a hot area — seven open PRs touch `flow-run.ts`.
- **#578** rewrites the CLI summary line I also change (one hunk, one
7-line function, semantically
independent — resolution is keep-both), and its
`attachFailureDiagnostics` dereferences
`env.device.platform`. **It will need a null-device guard**, or every
failure in a device-less flow
silently degrades to `read-failed` diagnostics. Happy to rebase onto it
rather than the reverse.
- **#585** rewrites `resolveRunDevice` at the same seam and edits the
same description sentence.
- **#538** rewrites `flow-device.ts`. Worth noting it makes
`resolveFlowDevice` reject a physical
iPhone for flows — skipping resolution means a narration-only flow now
passes on a
physical-iPhone-only host, which I think is desirable but is a second
behaviour change on this seam.
All new tests are in new files, since every existing flow test file in
the blast radius is rewritten
by an open PR.
## Behaviour changes a reviewer should weigh
- A device-free flow no longer prints `PASS on <id>` **even when a
device is booted**. That is the
point: otherwise an identical flow reports differently in CI (no
simulator) than on a laptop with
one open. An explicit `--device` is still honoured and still named.
- `--platform` is ignored for a device-free flow, where it previously
errored with nothing booted.
- A 0.18.1 CLI against a newer server would print `Flow "x" on ` —
cosmetic version skew.
## Verification
Live against the built CLI: device-less flow with nothing booted (the
reported bug) and with a device
booted (identical output, no ambient attribution); `--json` showing
`device: ""`; explicit `--device`
still naming it; a mixed flow and a composed flow both still demanding a
device; a mixed flow with a
device booted still reporting `PASS on <id> — 1 passed`.
24 new tests. tool-server 3103 passed, CLI 284 passed, MCP 78 passed —
no existing test modified.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
> **#697 is stacked on this branch** — it conflicts with this one in
`flow-run.ts`'s import block, so it was rebased on top. Merge this
first.
Fixes#617.
## Reproduced three ways
**The reported false green**, using an image whose 2× downscale is
lossless so no resampling noise can mask it:
```
baseline 800x1600 current 400x800 (exact half)
→ status: unchanged pixel_mismatch: 0% (size difference never mentioned)
```
**Worse on real screenshots** — a 1152×648 capture vs its
aspect-preserving half:
```
→ status: changed pixel_mismatch: 0.35% changed_areas: 8
Text changes: Disappeared "General" / Appeared "Remotes and Devices"
```
The content is identical; every region and both OCR "text changes" are
resampling artifacts. So the tool doesn't merely stay silent about the
size difference — it manufactures a plausible-looking regression report
out of noise.
**The control shows the asymmetry is the bug** — aspect-*breaking*
correctly reports `dimension_mismatch: expected=1,152x648
actual=1,152x400`. Sizes are disclosed exactly when the tool refuses to
compare, and never when it silently rescales.
## The fix
`PngDiffResult` gains `sizeNormalization` (both original sizes plus the
size compared at), populated only when a rescale actually happened — so
its presence *is* the signal that the comparison carries resampling
uncertainty. The summary prints it directly under the status:
```
- status: resized_no_change
- size_normalized: baseline=800x1,600 current=400x800 compared_at=400x800
- the inputs share an aspect ratio but not a resolution, so the larger was downscaled before
comparing; any pixel or text differences below may include resampling artifacts, and the diff
images are at compared_at rather than full size
- re-capture the baseline at the same scale as the current image (screenshot with scale: 1.0)
to compare without resampling
```
**`resized_no_change` is checked last**, after the difference checks, so
a rescaled comparison that *did* find something is still a plain
`changed` — a caveat status must never soften a real regression. There's
a test pinning that ordering specifically.
**The name matters.** `unchanged_after_resize` contains `"unchanged"`,
so the one naive gate this protects — `summary.includes("status:
unchanged")` — would still read it green. `resized_no_change` contains
neither `"unchanged"` nor `"changed"`.
## What I deliberately did not do
**Report `dimension_mismatch` whenever sizes differ** — the issue's
primary suggestion. The normalization is deliberate:
`screenshot-diff/index.ts:292-294` states that same-aspect normalization
*"keeps a scaled capture diff-compatible with a baseline saved at any
scale"*. The `screenshot` tool defaults to **0.3** for iOS/Android while
the diff's live capture is full-res, and on Android the live side falls
back to the default scale when full-res capture fails. Hard-failing
would break the case this code exists for.
**Expose the internal `normalizeSizes` flag.**
`screenshot-diff-tool.test.ts:10-23` is titled *"rejects public tuning
options so defaults stay internal"* and asserts exactly this kind of
param is refused. And the better remedy is the one the summary now gives
— re-capture the baseline at the same scale, which the skill already
instructs and which removes the resampling rather than reporting it.
Worth reconsidering separately if a user actually wants "green means
byte-identical".
*(An earlier draft proposed exposing it via `z.coerce.boolean()`. Worth
recording why that was dropped beyond the posture conflict: in zod,
`z.coerce.boolean("false")` is **`true`** — so a user asking for strict
comparison would have silently got normalization anyway, re-creating
this exact bug through the fix's own escape hatch.)*
**Suppress the OCR noise.** Text analysis runs on each *original file at
its own resolution* and only rescales the resulting bounds, so a
diff-time filter cannot separate "OCR read a 576px image badly" from a
real text change. Suppressing it whenever normalization occurred would
silence text analysis on every scaled comparison — a new, worse false
green of exactly the class this issue is about. Disclosure is the honest
treatment; running OCR on the normalized pair is a separate change to
`text-diff`'s contract.
## Tests
Three new cases plus three strengthened ones: the false green now
discloses and no longer reads `unchanged`; a normalized comparison with
a real change still reports `changed` *and* still discloses; equal sizes
say nothing; and the retina case still compares rather than
hard-failing.
Mutation-verified both halves — removing the status fails the #617
regression test; moving its check before the difference checks fails the
ordering test.
Full tool-server suite green — 3089 passed. No existing assertion needed
changing, which is also why this survived: nothing pinned it.
## Notes for the reviewer
- The flow snapshot path is untouched and **cannot** reach
normalization: full-screen baselines key on the capture's own pixel
dimensions (`flow-visual.ts:212`), so a differently-sized capture misses
the baseline entirely and fails; `cropOn` diffs already pass
`normalizeSizes: false`.
- `SummaryStatus` is unexported and nothing switches on it — the status
reaches the outside only as text inside `summary`, which the MCP layer
passes through verbatim. No skill doc, flow assertion or e2e script
matches on a status string.
- The diff artifacts were already produced at the normalized size; that
was simply never stated. The caveat now says so.
Fixes#634.
```
before: [Tool:chromium-tabs] Not supported
after: [Tool:chromium-tabs] Could not open a tab: Not supported. An Electron app has no
browser-level target creation, so a tab cannot be made from outside it — have the app open
one (e.g. `window.open()` through `debugger-evaluate`) and it will show up in
`chromium-tabs list`. The list, select and close actions are unaffected.
```
The old text is Electron's own CDP answer to `Target.createTarget`,
passed through untouched. The
limitation is real, but the message names neither the cause nor a way
forward, and reads as though the
tool were unavailable — when only this one action is.
## Two things I checked before writing the message
- **The workaround actually works.** On the Electron testbed I ran
`window.open('about:blank')` via
`debugger-evaluate` and the new window appeared in `chromium-tabs list`
as `t2`. I didn't want to
ship advice I hadn't run.
- **The rest of the tool really is fine** — `list` returns correctly
both before and after.
## Why the cause is kept rather than matched on
`Target.createTarget` can fail for reasons other than Electron's missing
browser shell — a dropped
connection, or a real Chrome refusing the url. The CDP client keeps only
the error's `message` and
discards its numeric code, so branching would mean matching on the
string `"Not supported"`, which
breaks the moment Electron rephrases it.
Instead any failure is wrapped, with the original message embedded. The
explanation is worded as a
property of Electron rather than a claim about *this* failure, so it
stays accurate when the cause is
something else — and the underlying error is still right there to read.
## Verification
Live against the Electron testbed (Electron 42 / Chrome 148): the
message above is the real output,
and `list` still works alongside it. tool-server 3087 passed, plus a new
test asserting the message
keeps the original cause *and* carries the reason, the workaround, and
the note that the other actions
are unaffected.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Problem
Field report: *"electron needs focus to be tested."* Analysis confirmed
three focus/visibility dependencies in the Chromium/Electron surface:
1. **Hidden window → throttled renderer.** A minimized/occluded window's
compositor throttles: mouse-input acks stall **~5 s per event** (they
wait on hit-testing), wheel scrolls hang (`gesture-scroll` already
refused), `visibilityState` flips to `hidden`.
2. **Unfocused window → focus-gated app logic breaks.**
`document.hasFocus()` is `false` before any input arrives;
focus/blur-dependent UI misbehaves.
3. Focus-gated web APIs (clipboard already worked around it locally with
`Page.bringToFront`).
## Fixes
**1. CDP focus emulation on every page session** (`primePageSession`, né
`enableCoreDomains`): `Emulation.setFocusEmulationEnabled(true)`,
best-effort, re-applied automatically after tab switches (Playwright
precedent). Live A/B on the Electron testbed: identical
boot→unfocus→probe sequence gives `hasFocus:false` on main, `true` on
this branch. Bonus verified by A/B (kill server → same minimized window
flips to `hidden`): in current Chromium/Electron, emulation also pins
reported visibility to `visible` and keeps input unthrottled while
genuinely minimized — minimized tap latency **~5 s → ~65 ms**, wheel
scroll works.
*Trade-off (same as Playwright ships):* while attached, the app can
observe neither real blur nor real hidden, so pause-on-hidden behavior
is unobservable during a test.
**2. Anti-throttling switches for every argent-booted Electron app**
(`--disable-background-timer-throttling
--disable-backgrounding-occluded-windows
--disable-renderer-backgrounding`). Sessions are created lazily and die
with the idle-exiting tool-server while the app outlives it — the flags
cover that detached lifecycle plus emulation-less runtimes. Also covers
flow-booted apps via the shared `bootElectronApp`.
**3. Hidden-window fail-fast extended from gesture-scroll to gesture-tap
+ gesture-drag** via shared `assertChromiumWindowVisible`, under a new
dedicated `CHROMIUM_WINDOW_HIDDEN` failure code (scroll migrated;
`failure_stage` strings unchanged so telemetry stays joinable —
analytics keyed on `CHROMIUM_INPUT_INVALID` +
`chromium_scroll_window_hidden` should switch to the new code). With
emulation active the guard is a backstop that fires only where emulation
couldn't be applied — exactly where the stall is real. Keyboard/button
stay unguarded on purpose: measured live, key events skip hit-testing
and stay fast on hidden windows.
## Verification
- Unit: 3075 tool-server + 87 registry tests green, `typecheck:tests`
green, new tests for emulation priming, boot args, and the guard (all
three tools).
- Live matrix on the Electron testbed (dev tool-server): focused
baseline; unfocused (`hasFocus` true pre-input, typing lands); minimized
booted app (tap 69 ms, scroll works, typing + slider drag land,
screenshot fine); minimized **external** un-flagged app on :9222 (tap 63
ms, scroll 460 ms — emulation alone suffices while attached).
- Plan + implementation went through three adversarial review rounds
(scope, CDP semantics, failure-code taxonomy/telemetry compat, doc
accuracy).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
> Recreates #543 — GitHub auto-closed it when the base branch of the
stacked PR (#542) was deleted on merge, and a closed PR cannot be
reopened after its head is force-pushed. Same commit, rebased onto main.
## Summary
**Was stacked on #542** (now merged). Consumes the
`ios.additionalDeviceSets` configuration across the iOS tool surface, so
simulators living in extra CoreSimulator device sets — e.g. **Radon
IDE's** `~/Library/Caches/com.swmansion.radon-ide/Devices/iOS` — are
discovered by `list-devices` and drivable by every tool, by UDID,
exactly like default-set devices.
### The core piece: a UDID → device-set map (`utils/ios-device-sets.ts`)
simctl scopes every operation to one device set, so each per-device call
must know the target's owning set. The new module owns that mapping and
the single argv choke point:
- **Learned from discovery**: `list-devices` and the simulator watcher
enumerate the default set + every configured set and `rememberDeviceSet`
each sighting — any tool used after a list hits a warm cache.
- **Lazy fallback**: an unknown UDID triggers one `simctl list` probe
per configured set, cached on hit. An unknown-everywhere UDID resolves
to the default set **without** caching (a device that appears later is
found on the next call) — preserving pre-feature behavior and error
messages byte-for-byte.
- **`simctlArgsForUdid(udid, args)`** builds the full argv (`--set`
injected only for additional-set devices); `simctlPrefix` covers
multi-call sites that resolve once; sync variants
(`simctlArgsForUdidSync`, `cachedDeviceSetForUdid`) cover the
`ChildProcess`-returning and `execFileSync` call sites, each of which
runs after an async step that warmed the cache.
- Config is **read off disk per call** (the #534 design): editing
`ios.additionalDeviceSets` applies without a tool-server restart.
- **Guard**: a configured set whose directory doesn't exist is never
queried — simctl would materialize the dir, turning a config typo into a
stray directory.
### Wired consumers
- **`list-devices`** — enumerates every set, tags additional-set devices
with `deviceSet`, documents the field.
- **`simulator-watcher`** — polls every set (a set failing to list
throws → the tick is skipped, so a transient error still can't
mass-dispose tracked services); warms the map for the injection path.
- **simulator-server blueprint** — passes `--device-set` (the same flag
Radon IDE passes to this shared binary); default-set spawns unchanged.
- **`ax-prefs`** — the pre-boot AX plist now lands under the owning
set's device dir (previously hardcoded to the default
`CoreSimulator/Devices` root, so the write would land nowhere for these
devices and describe would degrade).
- **`boot-device`** — set-aware shutdown/boot/bootstatus; skips the
Simulator.app attach + `CurrentDeviceUDID` write for additional-set
devices (one Simulator.app instance displays one device set — these run
headless and are streamed, exactly how Radon IDE presents them).
- **Every remaining per-device simctl call site**: launch-app /
restart-app / reinstall-app (both the direct impls and the
`SimctlBackend` shared handlers), open-url, settings-permissions
(privacy + install probe), the tvOS screenshot fallback, status-bar
pin/restore, device-shutdown, the ios-host `simctl spawn` family behind
describe/native-devtools, tv-control's AX daemon, and the native
profiler's sync helpers (cache warmed at `startNativeProfilerIos`).
**Out of scope**: `sim-remote` — the remote orchestrator host has its
own device-set story.
## Testing
- **Unit**: new `ios-device-sets.test.ts` (mapping, lazy probe + caching
semantics, no-probe fast path with no sets configured, discovery
enumeration/tagging/dedup, missing-dir guard); blueprint test asserts
the `--device-set` argv; the profiler shell-injection mock extended for
the new import. Full tool-server suite green: **276 files / 2752
tests**; `tsc --build`, eslint, prettier clean.
- **Live, end-to-end on a real Radon IDE device** (iPhone Air, iOS 26.4,
in `com.swmansion.radon-ide/Devices/iOS`), driving this branch's
tool-server over HTTP with the config set at project scope:
- `list-devices` → the Radon device appears, tagged with its `deviceSet`
(invisible to argent 0.16.1 on the same machine);
- `boot-device` → boots headless in its set, no Simulator.app window;
- `screenshot` → real home-screen PNG via simulator-server spawned with
`--device-set`;
- `launch-app` (Settings) → launched via `simctl --set launch`;
- `describe` → full accessibility tree from the Settings screen, i.e.
the AX bootstrap (set-aware prefs + in-sim ax-service daemon via `simctl
--set spawn`) works;
- clean shutdown; default-set devices meanwhile listed and behaved
exactly as before.
## Summary
Adds tool-owned, human-readable interaction messages for every tool
definition.
Each tool defines its own started, completed, and failed wording (as
implemented in https://github.com/software-mansion/argent/pull/367.
Messages can use validated parameters, successful results, and safe
failure codes. They are to avoid sensitive input such as typed text,
cookie values, JavaScript expressions, and URL credentials.
Bridges the gap between current Argent
<img width="954" height="493" alt="Screenshot%202026-07-28%20at%2015 48
37"
src="https://github.com/user-attachments/assets/2c689a93-8ba8-4208-b35e-66eb53c96671"
/>
and current agent-device
<img width="1188" height="476" alt="Zrzut ekranu 2026-07-28 o 20 14 42"
src="https://github.com/user-attachments/assets/6aed0f38-625b-4748-83e5-d384e79266c2"
/>
## Test
Selected fields from the generated JSONL:
```jsonl
{"level":30,"type":"tool.invoked","toolId":"list-devices","toolInvocationId":"decb864f-51c1-41ee-91fc-15bc2494e268","msg":"Listing devices","time":"2026-07-29T10:37:19.896Z"}
{"level":30,"type":"tool.completed","toolId":"list-devices","toolInvocationId":"decb864f-51c1-41ee-91fc-15bc2494e268","durationMs":96.77333299999918,"msg":"Listed 22 devices and 0 AVDs","time":"2026-07-29T10:37:19.993Z"}
{"level":30,"type":"tool.invoked","toolId":"screenshot","toolInvocationId":"8504abb8-83ed-4490-8210-3355d6c638c9","msg":"Capturing screenshot","time":"2026-07-29T10:37:24.503Z"}
{"level":50,"type":"tool.failed","toolId":"screenshot","toolInvocationId":"8504abb8-83ed-4490-8210-3355d6c638c9","failureSignal":{"error_code":"SIMULATOR_SERVER_READY_EXITED","failure_stage":"simulator_server_spawn_ready","failure_area":"tool_server","error_kind":"subprocess","failure_command":"simulator_server","failure_exit_code":1},"durationMs":750.4639999999999,"msg":"Failed to capture screenshot: SIMULATOR_SERVER_READY_EXITED","time":"2026-07-29T10:37:25.254Z"}
{"level":30,"type":"tool.invoked","toolId":"screenshot","toolInvocationId":"63b938fc-3d86-4eb5-9972-a008474ffb01","msg":"Capturing screenshot","time":"2026-07-29T10:38:01.107Z"}
{"level":30,"type":"tool.completed","toolId":"screenshot","toolInvocationId":"63b938fc-3d86-4eb5-9972-a008474ffb01","durationMs":25.509208000003127,"msg":"Captured screenshot argent-screenshot-chromium-cdp-9222-1785321481132-81477.png","time":"2026-07-29T10:38:01.133Z"}
```
## What
Adds the CSS combinators to flow selectors, read off frames. Any action
or condition can now scope its matches by where an element sits relative
to another:
```yaml
- tap: { text: Delete, within: { id: profile-card } } # CSS "A B" — inside that container
- tap: { role: Switch, next: { text: Wi-Fi } } # CSS "A + B" — the nearest follower
- assert: { visible: { role: Button, after: { text: Danger zone } } } # CSS "A ~ B" — any follower
- assert: { hidden: { any: true, within: { id: empty-state } } } # CSS "*" — the universal selector
```
| Scope | CSS | Reads as |
| --- | --- | --- |
| `within: <sel>` | `A B` | the match's frame sits inside the scope
element's frame |
| `after: <sel>` | `A ~ B` | the match follows the scope element in
reading order |
| `next: <sel>` | `A + B` | as `after`, narrowed to the nearest follower
of each anchor |
Plus `any: true` — the CSS `*` universal selector, accepted only
alongside a scope and never beside `text`/`id`/`role`.
The child combinator `>` has deliberately **no analog**: flow trees
flatten to leaves under one synthetic root, so parent/child structure
does not survive to replay.
## How
- **Geometric, not ancestry-based.** Frames are what survives
flattening, so every scope is resolved from them by `findAll` in
`ui-tree-match`. The synthetic root never counts as a scope element, and
every relation requires a *distinct* node (nothing scopes itself).
- **Reading order is row-band aware.** An element follows the anchor
when it starts below the anchor's bottom edge, *or* shares its row band
and sits entirely to its right. That band rule is what makes `{ role:
Switch, next: { text: Wi-Fi } }` resolve the Wi-Fi row's own switch even
though the taller switch's frame starts a couple of pixels *higher* than
the label's — a raw top-y comparison gets that backwards. `followKind`
is the single classification both the follower test and the nearest-pick
grouping read, so they cannot disagree.
- **`next` is the union over anchors**, exactly as CSS `A + B` is: three
rows on screen give three switches, one per label.
- **Scopes compose and nest**, capped at `MAX_SELECTOR_SCOPES` (6)
counted across the whole relation *tree* — a size bound, not a depth
bound, because three relations branch and the runner's loose-alternative
expansion is exponential in the bare-string scopes it finds.
- **Flow-YAML only.** `await-ui-element`'s strict schema rejects
`within`/`after`/`next`/`any`, so a scope can never be silently dropped
on that path.
- Adding a fourth relation is a **compile error**: `RELATION_RESOLVERS`
is a `Record` keyed by the relation union, so a new scope cannot be
wired through parse/serialize/reporting and then ignored by the match
engine.
## Testing
- **Unit:** full tool-server suite green — **3006 passed / 1 skipped**.
New coverage in `ui-tree-match.test.ts`, `flow-utils.test.ts`,
`flow-selector-scopes.test.ts` (renamed from
`flow-within-selector.test.ts`, now covering all scopes) and a new
`flow-skill-docs.test.ts` that parses every YAML snippet in the skill so
the docs cannot rot. Each fix is mutation-checked: reverting it fails at
least one test.
- **Real-device E2E** (Pixel 3a, Bluesky on Metro 8081, via a branch
tool-server): three flows, 20 steps, all green — geometric ordering both
ways, two `text` asserts that pin `next`'s pick *by identity* out of
five candidates, empty-scope `hidden` semantics, and two observable taps
(`next`-scoped and `any: true`-scoped) that navigated to the *adjacent*
topic rather than the anchor's own.
- **iOS not covered.** Native-devtools does not attach on either booted
simulator in this environment, so *every* flow selector fails there — an
unscoped `{ text: "Video" }` fails identically. Verified environmental,
not caused by this branch: the globally-installed argent reports the
same, direct `SIMCTL_CHILD_DYLD_INSERT_LIBRARIES` injection also fails
to connect, and it survives a forced simulator reboot on both iOS 18.3
and 26.5. Worth re-running the iOS leg once that is healthy.
## Notes for reviewers
Two deliberate deviations from CSS, both documented in the skill:
- **`next` is looser than `+`.** Where `A + B` matches nothing unless
the immediate sibling is a `B`, `next` keeps looking and returns the
nearest match further on. That is what makes it survive the wrapper and
spacer nodes a flattened tree is full of — but a row *missing* the
control you asked for resolves to the next row's rather than failing.
Happy to switch to strict `+` semantics if the loud failure is
preferred.
- **"follows" is not transitive.** Against a tall anchor an element can
follow something that follows the anchor without following it directly,
so nesting `after` scopes is not the same as chaining `~`.
Other things worth a look:
- `nearestAfter` and `afterTester` are direct scans, not indexed. An
earlier sorted index was a *second* implementation of the nearest rule
and the two disagreed on frames near the tolerance; it also measured
2-13x slower than the scan on realistic shapes, with a prune that
provably never fired. The quadratic tail is documented with measured
numbers: 1.2 ms for a single-anchor `next` at the 12k-node tree cap,
~555 ms only when the anchor selector matches essentially every node.
- A bare-string *anchor* cannot self-correct the way the loose fallback
usually does — a decoy `testID` matching your anchor text wins the
identifier pass whenever anything sits after it. The skill tells authors
to spell sibling anchors as maps.
- `MAX_SELECTOR_SCOPES = 6` replaces the previous
`MAX_SELECTOR_WITHIN_DEPTH = 10`, so a `within` chain deeper than 6 that
parsed on the earlier commits of this branch now rejects. Nothing
released depends on it.
---------
Co-authored-by: Hubert Gancarczyk <claude-hubert.gancarczyk@swmansion.com>
`snapshot` gains an optional `cropOn: <selector>` — diff one element's
region instead of the whole screen:
`- snapshot: { name: cart-total, cropOn: "Order summary" }`
- The selector resolves like a directive target: settled tree,
auto-wait, the standard
not-found failure. Selector-only — no point form (a point has no extent
to crop to).
- The frame resolves **before** the capture, and the **cropped** image
is what gets
compared, stored as the baseline, and reported as the `current`
artifact. The pixel
rect is the normalized frame × capture size; the crop is a pngjs
`bitblt` written to
a swept scratch dir and registered only when a failure returns it.
- The baseline key stays on the **full** capture's WxH (device-class
identity), so the
previously unreachable `dimensionMismatch` guard becomes load-bearing:
an element that
grew or shrank fails with "nothing was compared" plus a
crop-fixed-size-containers
hint. Deliberately strict — a 1px layout drift fails on dimensions, not
content (the
same tradeoff as Playwright element screenshots).
- A tree-source outage while resolving `cropOn` errors the step instead
of degrading to
a full-screen compare against a cropped baseline.
- No tool-schema changes; `snapshotKey` and `--output` behavior ride
along unchanged.
Tests: parse/serialize round-trip + rejection cases, and runSnapshot
coverage with real
PNGs (crop dims, key vs. content, artifact retention/sweep, unresolved
selector, abort,
size-drift message, sub-pixel region).
# feat(flow): rotate directive
Adds a selector-targeted two-finger rotation directive to flows:
`rotate: { on?, by }`. `on` resolves like `tap` (settled tree +
auto-wait); omitted, the rotation centers on the screen. `by` is
degrees, positive = clockwise. Distinct from `tool: rotate`, which
changes device orientation — the docs and step labels call this out.
## Design
**One continuous gesture, fixed pace, no duration option.** Fingers
orbit at ~90°/300 ms; duration derives from the angle. Unlike pinch
there is no sub-gesture chaining — any angle dispatches as one sweep, so
the angular delta is exact with no settle delays. `by` is parse-bounded
to ±3000°: the largest sweep the 10 s single-gesture envelope delivers
at that pace, derived from the constants rather than picked as policy —
rejected at parse, never silently clamped. A consequence embraced here:
frame count always scales with the angle, so a speed-sensitive recording
stays a raw `tool:` step (the create-flow skill's polish list covers the
conversion).
**Fixed centroid, constant physical radius.** Drifting the centroid
would couple pan into the turn; growing the radius would couple pinch.
Both stay fixed — the trade is a smaller orbit near edges and small
targets, in exchange for a pure rotation.
**Physical circle, not a normalized-space ellipse.** A single normalized
radius on a non-square screen traces a physical ellipse, coupling an
unintended pinch into every turn. `gesture-rotate` grows optional
per-axis `radiusX`/`radiusY` (fractions of width/height with `radiusX·W
= radiusY·H`), and the flow tree adapters surface screen dimensions via
an optional `screen` field on `DescribeTreeData` so the directive can
compute the aspect. Missing dimensions fail soft to the legacy
single-radius call rather than erroring.
**Placement reuses pinch's edge model.** The initial finger axis
(horizontal or vertical) is scored against the same per-platform
system-edge guards, preferring a start parallel to the nearest violated
edge and staying inside the target's axis-matching span. Guards only
score placement — they never shrink the radius, and a violated guard is
attempted, never rejected. The only geometry failure is no positive
on-screen orbit radius.
**Abort correctness.** `gesture-rotate` now honors `ctx.signal`: each
frame checks it, lifts the synthetic fingers with a terminal Up (so a
cancelled run doesn't leave them held on the glass), then throws with
name `AbortError`; the directive maps a mid-gesture abort to the run's
aborted skip rather than a step failure.
**Explicit `inputSchema`.** The radius requirement is a zod `.refine()`
cross-field rule, which `z.toJSONSchema` drops — so the tool ships a
derived schema plus an explicit `anyOf` (`radius` or
`radiusX`+`radiusY`) to keep invalid calls rejected at the schema layer.
**Platform rejections.** Vega is remote-driven (no touch) and Chromium
has no rotate-gesture idiom at all — both fail upfront with guidance to
drive the app's rotate controls directly.
## Tests
Unit tests cover the geometry module (duration derivation, candidate
construction, aspect handling, edge-guard ranking), the directive
end-to-end against a fake device, and `gesture-rotate`'s per-axis radii
and abort path.
## What
Draws the touch gestures argent drives directly into the screen
recording, so a captured video shows **where** each interaction landed
instead of a UI reacting to an invisible finger:
- a pulse marks each **tap**
- a fading comet trail follows a **swipe / drag**
- paired markers show the two contact points of a **pinch / rotate**
On by default; pass `showTouches: false` to `screen-recording-start` for
a raw, overlay-free capture.
## How
simulator-server already has a touch visualizer behind its `pointer`
cargo feature (the binary argent bundles ships with it). It renders the
markers into the very MJPEG stream this recording pipeline already
consumes — nothing is composited host-side, and it's driven by the touch
events argent already sends. This PR just turns it on for the life of a
recording:
- `screen-recording-start` builds a small `PointerControl` from the
resolved simulator-server handle and hands it to `startCapture`.
- The overlay is enabled right after the encoder goes live (so the first
interaction is already drawn) and **restored to off** when the capture
ends — stop, time-limit cap, encoder crash, or session teardown all run
the same disable path, so it never outlives the recording it was enabled
for.
- Two thin client helpers (`setPointerVisible`, `setPointerTrail`) POST
to simulator-server's `/api/pointer`. They're best-effort: a build
without the feature (or a remote sim) just yields a `warning` at stop
and a normal overlay-free video, never a failed recording.
## Verification
- **Unit** — `screen-recording.test.ts`: 49 pass, incl. 10 new covering
enable-at-start / disable at stop, cap, unexpected exit, and dispose;
the `pointerFailed` warning; the no-control path; and the `/api/pointer`
wire format (`{show}` / `{trail}`, ok/error/throw handling).
- **Static** — `tsc` (src + `tsconfig.test.json`), eslint, prettier,
full workspace build all clean.
- **E2E** — drove the real tool-server → the bundled (production)
simulator-server on an iOS 18.5 sim:
- `showTouches: true` + a slow swipe → a translucent pointer bubble is
rendered onto the screen at the touch point in the recording.
- identical swipe with `showTouches: false` → paging still happens, **no
bubble** — proving both the toggle and that stop restored the overlay to
off (the control clip never re-enables it, yet draws nothing).
---
Stacked on #548 (static-frame trimming) → base branch
`feat/screen-recording-trim`. Review after that merges.
---------
Co-authored-by: Ignacy Łątka <claude-hubert.gancarczyk@swmansion.com>
> Stacked on #517 (`feat/screen-recording`). Review/merge that first;
this PR's diff is against that branch.
## What
Trims stretches where the screen doesn't change out of the recorded
video. A recording left running across waits, slow navigation, or agent
thinking time used to come back padded with dead air; now a long session
with only brief activity comes back as a short clip.
## How
The frame pump already re-emits the last frame on a wall-clock schedule
to keep the timeline honest. This adds a change detector on top:
- Each frame is compared to the last one **byte-exact** (a reference
check short-circuits the common "no new frame arrived" case, so only
genuine arrivals pay the compare). Exact byte equality is stronger than
a hash — no collisions — and flags a change down to a single pixel,
matching "something moved, even by a couple of pixels."
- The first second of every still stretch is kept (pauses read
naturally), then unchanged frames are dropped until the screen moves
again.
- Active stretches still play at real time: the wall-clock pacer is
re-anchored whenever a dead stretch is skipped, so the gap yields no
output frames rather than a catch-up burst of the next frame.
On by default. `trimStatic: false` on `screen-recording-start` keeps a
faithful real-time recording (e.g. to measure how long something took on
screen). `screen-recording-stop` now returns the trimmed length as
`durationMs`, plus `wallClockMs` and `trimmedMs` when trimming removed
anything.
## Verification
- **Unit**: 39 tests in `screen-recording.test.ts` (7 new trim cases:
default-on, grace-window kept, no-trim control,
resume-the-moment-the-screen-changes,
byte-identical-new-frame-stays-trimmed, duration reporting,
fields-omitted-when-off). Full tool-server suite green (2817). `tsc
--build`, `typecheck:tests`, eslint, prettier clean.
- **E2E on a live tool-server (iOS 18.6 sim)**:
- 23s idle-heavy session (motion → 15s idle → motion) → **1.47s** clip,
`trimmedMs` 21930, with the real transitions intact (44 distinct frames
— above the 30-frame grace floor, proving post-grace motion is kept). A
`trimStatic:false` control of the same pattern returned the full 23s /
691 frames.
- 8s continuous burst-motion session → **2.1s** across multiple motion
bursts (not collapsed to the 1s floor).
- **Cross-check**: ran ffmpeg's own `mpdecimate` (duplicate-frame
detector) over the same untrimmed footage — it kept ≤ what this trim
keeps, confirming the trim is faithful, never over-aggressive
(byte-exact is deliberately more conservative than mpdecimate's
near-duplicate heuristic, so smooth animation is preserved while only
frozen stretches are dropped).
## What
Adds a `screen-recording-start` / `screen-recording-stop` tool pair
(naming follows `native-profiler-start/stop`) that records the device
screen to an mp4 and returns it as a downloadable artifact.
Capture runs through **simulator-server** — the same backend
`screenshot` and every interaction tool already drive. simulator-server
publishes the device screen as an MJPEG stream (`/stream.mjpeg`), so
recording needs no platform-specific recorder: the tool-server
subscribes to that stream, paces the frames onto a fixed 30 fps
timeline, and pipes them into a single ffmpeg process that encodes — and
watermarks — straight to the final mp4. The stream is multi-subscriber,
so a preview window can stay open on the same device while recording.
- **One path for every device simulator-server drives** — iOS
simulators, Android emulators, and physical Android devices.
- **Native resolution, constant 30 fps** — h264, `+faststart`, no
post-processing pass.
- Session state lives in a per-device `ScreenRecordingSession` blueprint
(mirrors the native-profiler session): time-limit cap with
auto-finalize, unexpected-exit recovery (a capture that ended on its own
is still retrievable via stop), always-startable state reset on stop,
and dispose-on-shutdown cleanup through the registry.
## Why the frames are paced host-side
A device only emits a frame when the screen **changes**. Pointing ffmpeg
straight at the stream therefore produces a video whose length is the
sum of the moments something moved — a still screen collapses to a
fraction of a second — and an ffmpeg blocked on a silent socket does not
answer a stop signal promptly either (measured: 12 s late). Re-emitting
the last frame on a wall-clock schedule fixes both: **video duration
always equals real elapsed time**, and stop is a stdin EOF that
finalizes in ~60 ms. Duplicate frames cost almost nothing once encoded
(a 10 s recording of a static screen is ~1 MB).
## Finishing pass folded into capture
The Argent watermark (logo + `By @swmansion`, bottom-left, sized
relative to the frame) is stamped by the same ffmpeg invocation that
encodes the capture, so there is no second encode and no temp
intermediate. Its colour is chosen **per pixel** from the background
luma underneath it — the white logo over dark areas, a runtime-derived
near-black copy over light areas — so it keeps maximum contrast against
whatever is behind it. It sits at 20 % opacity and is opt-out via
`argent disable video-watermark` (read when the recording starts). Frame
geometry comes from the first JPEG's SOF header, so no `ffprobe` is
needed.
ffmpeg is now the recorder rather than an optional finisher, so a
missing ffmpeg fails `screen-recording-start` up front with an install
hint instead of silently degrading the output at stop time.
## Reminder plumbing
While a recording is running — or ended on its own but was never
retrieved — **every** tool result carries a `note` telling the agent the
recording is still going and how to stop it. It rides the same `note`
field as the update reminder but is per-call and unsuppressed; both
notes compose when they coincide. The note text switches to "already
ended, retrieve the file" once the cap fires or the capture dies.
## Skill
New `argent-screen-recording` skill (+ routing stanza in
`rules/argent.md`): documents the start → interact → stop workflow and
instructs the agent to schedule its own reminder/wakeup for the expected
end of the recording immediately after starting one (harness reminder
tool if available, otherwise a background `sleep` whose completion
notification pulls it back). Scores 10/10 on `scripts/grade-skills.mjs`.
## Verification
- Unit: 45 tests across `screen-recording.test.ts`,
`mjpeg-stream.test.ts`, `watermark.test.ts`,
`http-recording-note.test.ts`; full tool-server suite **2808 tests / 280
files green**; `tsc --noEmit -p tsconfig.test.json`, root `tsc --build`,
eslint, prettier all clean.
- E2E on a live tool-server (`POST /tools/...`), driving real
interactions between start and stop:
- **iOS sim** (iPhone 16 Pro Max, iOS 18.5): 1320×2868 h264,
`r_frame_rate=30/1`, 338 frames, container duration 11.27 s vs reported
`durationMs` 11314 — **within 50 ms of wall clock**. Extracted frames
show real screen content (mid app-switcher animation) with the watermark
rendered light-over-dark.
- **Android emulator** (Pixel 3a, API 34): 1080×2220 h264, 30 fps, 353
frames, duration 11.77 s vs `durationMs` 11793 — within 30 ms. 512 KB
for a mostly-static home screen.
- **Static screen** (no interaction at all, 10 s): 300 frames, duration
exactly 10.000 s — the case the previous `recordVideo` implementation
reduced to ~0.03 s.
- **Long silence then activity** (15 s untouched, then an app launch):
stream survived, no warning, frames after the silence show the new
screen. This caught a real bug during review — the connect timeout was
also acting as a socket *inactivity* timeout and tore the stream down at
10 s of stillness; fixed and covered by a regression test that fails
without the fix.
- **Time-limit cap**: `timeLimitSeconds: 5` self-finalized at 4.97 s /
149 frames, the per-call note flipped to "already ended (it hit its 5s
time limit)", and a stop issued 11 s later still returned the video with
the matching warning.
## Not supported (capability-gated with clear errors)
tvOS simulators, physical iPhones, Chromium apps, Vega, and remote
(`remote:`-prefixed) simulators — none of them expose a readable frame
stream.
---------
Co-authored-by: Ignacy Łątka <claude-hubert.gancarczyk@swmansion.com>
## Problem
`native-profiler-stop` fails with `No active native profiling session
found. Call native-profiler-start first.` while xctrace is still
recording — #248, #250, #330. Those were closed on the 0.12.1
`--all-processes` workaround (#380), which fixed trace finalization but
not this.
The tool isn't declared `longRunning`, so the MCP adapter applies its
per-request timeout:
```js
fetchTimeoutMs: meta?.longRunning ? null : FETCH_TIMEOUT_MS, // mcp-server.ts:222
```
A stop that runs past 30s is aborted and retried (`MAX_RETRIES = 4`).
The retry runs against the session state the first call has already
cleared, hits the `!api.profilingActive` branch and throws. The message
points at session tracking; the actual failure is the timeout.
Every failed stop I have logs for lands at 30–34s: `durationMs: 33639`
in [#330](https://github.com/software-mansion/argent/issues/330), and
30276 / 30605 / 31019 / 31914 locally.
Stop can't fit in 30s on Xcode 26.4+. Since #380 the capture is
host-wide rather than app-scoped, so the bundle is much larger — ~55MB
per 10s of recording here, 385MB for 90s — and xctrace needs ~22s to
package even the 10s case before the four export passes start.
`STOP_GRACE_MS` has the same problem from the other side: at 30s xctrace
is SIGTERM'd mid-package, and the bundle it leaves behind fails to
export.
## Fix
Declare the tool `longRunning`, and raise `STOP_GRACE_MS` to 300s. The
grace is a ceiling rather than a fixed wait — a fast package still
returns as soon as xctrace exits — so this only changes captures that
were previously killed.
With both applied a 90s capture stops and exports cleanly in ~2–4 min,
and a 40s capture returns inline.
## Note
The abort-and-retry sequence is inferred rather than captured directly.
What's observed: `MAX_RETRIES = 4`, the uniform ~30s durations, and
`tool-server.log` showing the `did not respond to SIGINT` warning —
written after the throw site — logged 184ms *before* the failure was
returned. One execution can't produce that ordering.
## Testing
- `npm run build` and eslint clean on changed files
- `npm test -w @argent/tool-server` — 2727 tests / 273 files pass
- Verified on Xcode 26.5 / iOS 26.5 simulator against a bridgeless RN
app: start → interact → stop → analyze now returns a populated report
with `errors: {}`
Adds a selector-targeted pinch-zoom directive to flows: `pinch: { on?,
scale }`. `on` resolves like `tap` (settled tree + auto-wait); omitted,
the pinch centers on the screen. `scale > 1` zooms in, `< 1` zooms out.
## Design
**Options-map only, no bare form.** `tap: "Map"` is unambiguous; `pinch:
"Map"` isn't (in or out?), so the parser requires `{ on?, scale }` and
rejects a bare selector with authoring guidance.
**Validity checks, not policy caps.** `scale` must be finite, > 0, and ≠
1 (a no-op scale is almost certainly a mistake) — but there is
deliberately no magnitude cap. An extreme scale decomposes
logarithmically into equal-ratio sub-gestures of ≤4× each, chained with
a 250 ms recognizer-reset delay (`scale: 1e6` is just 10 gestures).
**Pure geometry module.** All placement math lives in
`flow-pinch-geometry.ts` as deterministic data-in/data-out functions (no
device access), so tests drive it with synthetic centers, spans, and
edge guards. Per sub-gesture, a horizontal and a vertical candidate are
built from the axis-matching target dimension (normalized X and Y are
different units, so never `min(width, height)`), and the better one is
dispatched.
**Edge safety scores, it never rejects.** Per-platform system-edge
guards (wider on Android's back-gesture zones) demote candidates in
ranking but never fail a step: viable travel beats edge safety (a
risky-but-perceptible gesture beats a safe no-op the recognizer
ignores), then fully-safe with the most travel, then motion parallel to
the violated edge (edge swipes need motion *away* from the edge). Guards
are resolved once per directive and passed as data — the seam for a
future per-device query. The only geometry failure is literally zero
on-screen finger travel; a tiny target is still attempted.
**Centroid drift near edges.** When expanding fingers would leave the
screen, the centroid drifts inward — only on the moving axis, and only
as far as needed. `gesture-pinch` grows optional
`endCenterX`/`endCenterY` to interpolate the centroid alongside
separation; omitted, behavior is unchanged.
**Open-loop by design.** There is no "current zoom" to read back, so the
directive doesn't try to verify the multiplier — flows assert on the
resulting UI instead.
**Platform rejections.** Vega is remote-driven (no touch input) and
Chromium has no uniform pinch-zoom mapping for a desktop app (ctrl+wheel
or app-specific controls) — both fail upfront with guidance to drive the
app's zoom UI directly.
## Tests
Unit tests cover the geometry module (decomposition, candidate
construction, edge-guard ranking), the directive end-to-end against a
fake device, and the `gesture-pinch` centroid-drift interpolation.
## Summary
Fixes **bucket B** of the 0.16.0 crash telemetry — the `toolserver_bind`
/ `EADDRINUSE` restart loop. It's the largest crash **volume** (~817
events, one machine alone = 785) even though it's a narrow user set
(10). Companion to #531 (which fixed the widespread multi-user bucket
A).
## Root cause (reproduced live)
The main HTTP listener's `server.on("error")` handler routed **every**
bind error through `crashShutdown` (exit 1). When a second tool-server
starts on an already-owned port, that exit just feeds a supervisor
respawn → re-bind → crash, forever.
The telemetry signature was puzzling: all these crashes are
`crash_phase=serving` with ~1s uptime and **0 tool calls** — i.e. the
listener *bound successfully* and then died. I reproduced it: on macOS,
a **second tool-server process's** `listen()` momentarily **succeeds**
(the `listening` callback runs → `listening=true` → phase `serving`),
and **then** the server emits an asynchronous `EADDRINUSE`. So the
redundant instance binds, immediately errors, crashes, and the
supervisor loops it (~1s each, 785× on one machine).
Live A/B on the compiled `dist` (isolated port, telemetry disabled),
server A up then a second server B on the same port:
```
buggy (current 0.16.0): B: "Tools server listening ..." → "Failed to bind ... EADDRINUSE" → exit 1 (→ respawn loop)
fixed: B: "Tools server listening ..." → "deferring to it (redundant instance)" → exit 0
```
A survives in both.
## Fix
Handle `EADDRINUSE` specifically instead of crash-looping:
- Probe the port for a healthy argent tool-server (`GET /tools` → 200,
or 401 when a token is configured).
- **If a healthy peer answers**, this instance is redundant → shut down
**cleanly** (`shutdown(0)`) and defer to it. This removes it from the
`reason:"crash"` restart-loop population entirely.
- **If nothing healthy owns the port** (a foreign holder, or a wedged
peer that doesn't answer within the probe timeout) → still crash, so a
genuinely stuck port stays visible.
- `EACCES` and all other server errors crash exactly as before.
It also corrects the now-false comment claiming these crashes are phased
`startup` ("the listen callback never ran"). Telemetry and the live
repro prove the bind succeeds first, so `listening=true` and the phase
is genuinely `serving`.
## Testing
- **New unit tests** for `probeArgentToolServer`: peer answering 200 or
401 → `true`; foreign 500, connection-refused, and wedged/timeout →
`false`.
- Full tool-server suite **2731 green**; `tsc` + `eslint` clean.
- **Live A/B** above, on the built dist.
## Note on scope
This is a robustness fix that breaks the loop for the redundant-instance
case (the dominant one). The precise reason macOS lets a second
process's bind succeed-then-error asynchronously (vs. failing
immediately, as same-process contention does) is a kernel/loopback
timing detail; the fix does not depend on it — any `EADDRINUSE`,
immediate or deferred, now defers-or-crashes deterministically rather
than hot-looping.
Jakub Piasecki