callstack__agent-device/scripts/maestro-conformance/README.md

# Maestro conformance oracle

A three-layer oracle that proves the private agent-device Maestro package (`packages/maestro`)
stays faithful to a version-pinned upstream Maestro. It replaces the original
hand-typed parser fixture, whose transcribed expectations let four bug classes
slip through during #1217. Every expected value here is **generated from the
pinned upstream artifacts** — hand transcription is the failure mode this
replaces (issue #1274).

Pinned upstream: `dev.mobile:*:2.5.1` (`v2.5.1` / `a4c7c95f`), see
[`pinned-upstream.json`](./pinned-upstream.json).

## Layers

| Layer | What it proves | Generated by | Runs in |
|-------|----------------|--------------|---------|
| 1 — parser | Our parser accepts/rejects/normalizes each flow like upstream | `maestro-orchestra`'s `YamlCommandReader` over the corpus | `node --test`, per-PR |
| 2 — semantics | Our geometry/retry/timing constants match upstream | ASM-read bytecode constants + parser-observed model defaults | `node --test`, per-PR |
| 3 — differential | Outcome parity on a device, plus engine-side timing invariants | Real Maestro vs `agent-device test` | dispatch-only (see below) |

Layers 1–2 are checked-in generated fixtures (`fixtures/`) verified
deterministically with **no Java**. Layer 3 needs a device and the `maestro` CLI.

### How "generated from upstream" is enforced

Per-PR CI cannot re-derive the fixtures — that needs Java, and staying Java-free
on the normal path is a design constraint. So enforcement is two-layered:

1. **Per-PR:** every fixture carries a `contentHash` seal over its generated
   content, recomputed on each verify run. Editing a captured command or constant
   by hand breaks the seal. This is tamper-**evident** — it makes casual or
   accidental hand-editing impossible, but a determined editor could recompute it.
2. **Scheduled (`conformance-regenerate`):** actually re-runs the JVM harness
   against the pinned jars and fails if the checked-in fixtures differ by a byte.
   Forgery cannot survive a real re-derivation.

Without (2), "generated from upstream" would be documentation. Do not weaken
either half: together they are what stops this oracle from decaying back into the
hand-typed fixture it replaced.

### What layer 3 actually proves

Read `scenarios.ts` literally. Cross-engine comparison is **outcome parity** — it
only catches a divergence severe enough to fail the flow. It cannot see settle
latching, retap counts, or a 1px truncation difference. Where finer behavior
matters we assert it **engine-side** via `engineInvariants` over agent-device's
own `replay-timing.ndjson`.

Layer-3 flows live in `differential/flows/` and drive the **real fixture app**
(`examples/test-app`, `com.callstack.agentdevicelab`), which the workflow builds
and installs. They are deliberately **not** the layer-1 corpus: those flows exist
only to be *parsed* — they name a fictional `com.example.app` and elements that
exist on no device — so a device run against them would fail before exercising
any runtime behavior, making the settle detector silently vacuous. A test
enforces the separation, and the workflow hard-fails if the app is not installed.

### Declared divergences (`knownDivergence`)

Layer 3 has the same contract as layer 1: **every divergence is a decision on the
record.** When the differential catches a real engine bug, the instrument does not
block on repairing what it just measured — the scenario declares it with a
`knownDivergence: { reason, tracking }`, the scheduled run stays green on that
known gap, and only *undeclared* divergences fail.

Two rules keep that from rotting, both enforced by `run.test.ts` / the runner
rather than by good intentions:

- **`tracking` is required** and must be a real issue URL. A declared divergence
  with nothing behind it is how "temporarily expected" silently becomes permanent.
- **A stale declaration fails.** If a declared-divergent scenario starts passing,
  the run goes red until the declaration is removed — so the fix PR must delete
  it, and the oracle then enforces that the gap stays closed. The differential is
  the acceptance test for its own findings.

A green run still prints what it is *not* proving.

This matters most for **bug class 4** (settle ordering), whose 200ms × 10 loop has
no reflectable upstream constant, so layer 3 is its only home. Its detector is the
invariant "a tap must not consume the entire settle budget" — a full-budget tap
(~2093ms against 2000ms) means the stability loop never latched, yet the flow
still passes, so outcome parity would miss it. The evaluator is pure and
unit-tested against synthetic traces (`invariants.test.ts`); only the device run
that produces a real trace is scheduled-only. A scenario that declares an
invariant fails if the trace is missing — a detector that cannot run is a
failure, not a pass.

## Files

- [`jvm-harness/`](./jvm-harness) — Gradle/Kotlin generator for layers 1–2. Depends on the
  published Maestro jars; reads the parser and constants directly (never
  transcribed). Requires JDK 17+.
- [`corpus/`](./corpus) — flows driven through the upstream parser.
  `manifest.json` (generated by [`build-manifest.mjs`](./build-manifest.mjs), do
  not hand-edit) records provenance: upstream flows are vendored verbatim with
  their `sha256`; authored flows fill coverage gaps (`authored/`), encode the bug
  classes (`bug-classes/`), and give the never-accept-what-upstream-rejects guard
  teeth (`invalid/`). To add a flow: drop the `.yaml` in, add a note to `NOTES`
  in `build-manifest.mjs`, and regenerate.
- [`fixtures/`](./fixtures) — the generated, checked-in layer-1/layer-2 captures.
- `packages/maestro/src/internal/conformance-normalize.ts` — package-private canonical projection.
- `packages/maestro/test/conformance/` — the deterministic verifier, declared divergences,
  package-private harness, shared fixture seal, and layer-3 differential scenarios. Keeping this
  code under package tests prevents parser/canonicalization tooling from widening the production
  facade; regeneration imports the same package-owned seal implementation.
- [`regenerate.mjs`](./regenerate.mjs) — SHA-verifies the jars, rebuilds the
  corpus manifest and fixtures, and seals them.

## Verify (per-PR, no Java)

```sh
pnpm maestro:conformance
```

The verifier fails on any **undeclared** divergence: a flow our engine parses
that upstream rejects (a conformance regression), an undeclared parse mismatch,
or an undeclared `we-reject`. Every `we-reject` must list the unsupported
command/option in `packages/maestro/test/conformance/expected-divergence.ts` — that
list is the mechanical parity record. A focused issue is attached only when
implementation work is planned.

## Regenerate (only on an upstream-pin bump)

Heavy, manual, needs JDK 17+ (Gradle is provided by the committed wrapper):

```sh
pnpm maestro:conformance:regenerate
```

This resolves the pinned jars, **verifies their SHA-256 against
`pinned-upstream.json`**, runs the harness over the corpus, and rewrites
`fixtures/`. Review the diff, then run the verifier. To bump the pin: update
`pinned-upstream.json` (version/tag/commit + the jar SHA-256s from Maven Central),
refresh the vendored corpus flows and their `manifest.json` `sha256`s, regenerate,
and reconcile any new divergences.

## Layer 3 (device)

```sh
pnpm maestro:conformance:differential -- --platform ios --out-dir .tmp/diff
```

Runs scheduled on the `conformance-differential` workflow, and on demand via
`workflow_dispatch`. `--dry-run` validates the scenario registry without a device.

The workflow builds and installs the fixture app, verifies its bundle id, pins
the Maestro CLI to the same version as layers 1-2, and passes `--maestro` so the
flow routes through the compat engine.

The built `.app` is **cached** (keyed on the app's sources, its dependency graph,
the iOS runtime, and the Xcode version), because building it costs ~22 minutes
versus ~6 minutes for the differential itself — 79% of the job, for an app that
changes almost never. A cache hit installs the bundle directly; a miss falls back
to the full build and repopulates. If you change anything under
`examples/test-app`, expect the next run to rebuild.

**Investigate locally, not through CI.** A device iteration in CI is ~40 minutes;
`--only` plus a local simulator is minutes:

```sh
pnpm test-app:install && pnpm --dir examples/test-app exec expo run:ios --configuration Release
pnpm maestro:conformance:differential -- --platform ios --only settle-after-tap --trace-root .agent-device
```