Rohit Bansal
d11c8cf9d6
* feat: support standalone Maestro clearState command Accept '- clearState' / '- clearState: <appId>' in Maestro YAML flows. Unlike launchApp.clearState (clear-then-open), the standalone form clears app state without relaunching, projecting to 'settings clear-app-state' on the daemon. Covers the Rocket.Chat login-with-deeplink helper, which previously failed with 'Maestro command "clearState" is not supported'. * test(maestro): cover standalone clearState with authored corpus flow Replace the UNVERIFIED_COMMANDS exemption with an authored clear-state flow exercising default and explicit appIds, plus the regenerated upstream parser fixture proving Maestro compatibility. Live iOS Simulator evidence (iPhone 16, com.apple.mobilesafari): - marker files in the data container, then replay '- clearState' (default) and '- clearState: <appId>' (explicit) via 'replay --maestro'; both replay 1/1, wipe the container, and leave MobileSafari not running (no reopen).
Maestro conformance oracle
A three-layer oracle that proves the private agent-device Maestro package (packages/maestro)
stays faithful to a version-pinned upstream Maestro. It replaces the original
hand-typed parser fixture, whose transcribed expectations let four bug classes
slip through during #1217. Every expected value here is generated from the
pinned upstream artifacts — hand transcription is the failure mode this
replaces (issue #1274).
Pinned upstream: dev.mobile:*:2.5.1 (v2.5.1 / a4c7c95f), see
pinned-upstream.json.
Layers
| Layer | What it proves | Generated by | Runs in |
|---|---|---|---|
| 1 — parser | Our parser accepts/rejects/normalizes each flow like upstream | maestro-orchestra's YamlCommandReader over the corpus |
node --test, per-PR |
| 2 — semantics | Our geometry/retry/timing constants match upstream | ASM-read bytecode constants + parser-observed model defaults | node --test, per-PR |
| 3 — differential | Outcome parity on a device, plus engine-side timing invariants | Real Maestro vs agent-device test |
dispatch-only (see below) |
Layers 1–2 are checked-in generated fixtures (fixtures/) verified
deterministically with no Java. Layer 3 needs a device and the maestro CLI.
How "generated from upstream" is enforced
Per-PR CI cannot re-derive the fixtures — that needs Java, and staying Java-free on the normal path is a design constraint. So enforcement is two-layered:
- Per-PR: every fixture carries a
contentHashseal over its generated content, recomputed on each verify run. Editing a captured command or constant by hand breaks the seal. This is tamper-evident — it makes casual or accidental hand-editing impossible, but a determined editor could recompute it. - Scheduled (
conformance-regenerate): actually re-runs the JVM harness against the pinned jars and fails if the checked-in fixtures differ by a byte. Forgery cannot survive a real re-derivation.
Without (2), "generated from upstream" would be documentation. Do not weaken either half: together they are what stops this oracle from decaying back into the hand-typed fixture it replaced.
What layer 3 actually proves
Read scenarios.ts literally. Cross-engine comparison is outcome parity — it
only catches a divergence severe enough to fail the flow. It cannot see settle
latching, retap counts, or a 1px truncation difference. Where finer behavior
matters we assert it engine-side via engineInvariants over agent-device's
own replay-timing.ndjson.
Layer-3 flows live in differential/flows/ and drive the real fixture app
(examples/test-app, com.callstack.agentdevicelab), which the workflow builds
and installs. They are deliberately not the layer-1 corpus: those flows exist
only to be parsed — they name a fictional com.example.app and elements that
exist on no device — so a device run against them would fail before exercising
any runtime behavior, making the settle detector silently vacuous. A test
enforces the separation, and the workflow hard-fails if the app is not installed.
Declared divergences (knownDivergence)
Layer 3 has the same contract as layer 1: every divergence is a decision on the
record. When the differential catches a real engine bug, the instrument does not
block on repairing what it just measured — the scenario declares it with a
knownDivergence: { reason, tracking }, the scheduled run stays green on that
known gap, and only undeclared divergences fail.
Two rules keep that from rotting, both enforced by run.test.ts / the runner
rather than by good intentions:
trackingis required and must be a real issue URL. A declared divergence with nothing behind it is how "temporarily expected" silently becomes permanent.- A stale declaration fails. If a declared-divergent scenario starts passing, the run goes red until the declaration is removed — so the fix PR must delete it, and the oracle then enforces that the gap stays closed. The differential is the acceptance test for its own findings.
A green run still prints what it is not proving.
This matters most for bug class 4 (settle ordering), whose 200ms × 10 loop has
no reflectable upstream constant, so layer 3 is its only home. Its detector is the
invariant "a tap must not consume the entire settle budget" — a full-budget tap
(~2093ms against 2000ms) means the stability loop never latched, yet the flow
still passes, so outcome parity would miss it. The evaluator is pure and
unit-tested against synthetic traces (invariants.test.ts); only the device run
that produces a real trace is scheduled-only. A scenario that declares an
invariant fails if the trace is missing — a detector that cannot run is a
failure, not a pass.
Files
jvm-harness/— Gradle/Kotlin generator for layers 1–2. Depends on the published Maestro jars; reads the parser and constants directly (never transcribed). Requires JDK 17+.corpus/— flows driven through the upstream parser.manifest.json(generated bybuild-manifest.mjs, do not hand-edit) records provenance: upstream flows are vendored verbatim with theirsha256; authored flows fill coverage gaps (authored/), encode the bug classes (bug-classes/), and give the never-accept-what-upstream-rejects guard teeth (invalid/). To add a flow: drop the.yamlin, add a note toNOTESinbuild-manifest.mjs, and regenerate.fixtures/— the generated, checked-in layer-1/layer-2 captures.packages/maestro/src/internal/conformance-normalize.tsandconformance-selector-projection.ts— package-private canonical projection and selector model.packages/maestro/test/conformance/— the deterministic verifier, declared divergences, package-private harness, shared fixture seal, and layer-3 differential scenarios. Keeping this code under package tests prevents parser/canonicalization tooling from widening the production facade; regeneration imports the same package-owned seal implementation.regenerate.mjs— SHA-verifies the jars, rebuilds the corpus manifest and fixtures, and seals them.
Verify (per-PR, no Java)
pnpm maestro:conformance
The verifier fails on any undeclared divergence: a flow our engine parses
that upstream rejects (a conformance regression), an undeclared parse mismatch,
or an undeclared we-reject. Every we-reject must list the unsupported
command/option in packages/maestro/test/conformance/expected-divergence.ts — that
list is the mechanical parity record. A focused issue is attached only when
implementation work is planned.
Regenerate (on an upstream-pin bump or intentional corpus/harness change)
Heavy, manual, needs JDK 17+ (Gradle is provided by the committed wrapper):
pnpm maestro:conformance:regenerate
This resolves the pinned jars, verifies their SHA-256 against
pinned-upstream.json, runs the harness over the corpus, and rewrites
fixtures/. Review the diff, then run the verifier. A pin bump is not required
when a corpus flow or harness behavior intentionally changes; regenerate from
the unchanged pin and review the generated diff. To bump the pin: update
pinned-upstream.json (version/tag/commit + the jar SHA-256s from Maven Central),
refresh the vendored corpus flows and their manifest.json sha256s, regenerate,
and reconcile any new divergences.
Layer 3 (device)
pnpm maestro:conformance:differential -- --platform ios --out-dir .tmp/diff
Runs scheduled on the conformance-differential workflow, and on demand via
workflow_dispatch. --dry-run validates the scenario registry without a device.
The workflow builds and installs the fixture app, verifies its bundle id, pins
the Maestro CLI to the same version as layers 1-2, and passes --maestro so the
flow routes through the compat engine.
The built .app is cached (keyed on the app's sources, its dependency graph,
the iOS runtime, and the Xcode version), because building it costs ~22 minutes
versus ~6 minutes for the differential itself — 79% of the job, for an app that
changes almost never. A cache hit installs the bundle directly; a miss falls back
to the full build and repopulates. If you change anything under
examples/test-app, expect the next run to rebuild.
Investigate locally, not through CI. A device iteration in CI is ~40 minutes;
--only plus a local simulator is minutes:
pnpm test-app:install && pnpm --dir examples/test-app exec expo run:ios --configuration Release
pnpm maestro:conformance:differential -- --platform ios --only settle-after-tap --trace-root .agent-device