mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
Michał Pierzchała
f065e6aeb4
* fix(maestro): align label metadata ownership * fix(maestro): centralize command label parsing * test(maestro): cover runFlow label ownership
166 lines
8.7 KiB
Markdown
166 lines
8.7 KiB
Markdown
# Maestro conformance oracle
|
||
|
||
A three-layer oracle that proves the private agent-device Maestro package (`packages/maestro`)
|
||
stays faithful to a version-pinned upstream Maestro. It replaces the original
|
||
hand-typed parser fixture, whose transcribed expectations let four bug classes
|
||
slip through during #1217. Every expected value here is **generated from the
|
||
pinned upstream artifacts** — hand transcription is the failure mode this
|
||
replaces (issue #1274).
|
||
|
||
Pinned upstream: `dev.mobile:*:2.5.1` (`v2.5.1` / `a4c7c95f`), see
|
||
[`pinned-upstream.json`](./pinned-upstream.json).
|
||
|
||
## Layers
|
||
|
||
| Layer | What it proves | Generated by | Runs in |
|
||
|-------|----------------|--------------|---------|
|
||
| 1 — parser | Our parser accepts/rejects/normalizes each flow like upstream | `maestro-orchestra`'s `YamlCommandReader` over the corpus | `node --test`, per-PR |
|
||
| 2 — semantics | Our geometry/retry/timing constants match upstream | ASM-read bytecode constants + parser-observed model defaults | `node --test`, per-PR |
|
||
| 3 — differential | Outcome parity on a device, plus engine-side timing invariants | Real Maestro vs `agent-device test` | dispatch-only (see below) |
|
||
|
||
Layers 1–2 are checked-in generated fixtures (`fixtures/`) verified
|
||
deterministically with **no Java**. Layer 3 needs a device and the `maestro` CLI.
|
||
|
||
### How "generated from upstream" is enforced
|
||
|
||
Per-PR CI cannot re-derive the fixtures — that needs Java, and staying Java-free
|
||
on the normal path is a design constraint. So enforcement is two-layered:
|
||
|
||
1. **Per-PR:** every fixture carries a `contentHash` seal over its generated
|
||
content, recomputed on each verify run. Editing a captured command or constant
|
||
by hand breaks the seal. This is tamper-**evident** — it makes casual or
|
||
accidental hand-editing impossible, but a determined editor could recompute it.
|
||
2. **Scheduled (`conformance-regenerate`):** actually re-runs the JVM harness
|
||
against the pinned jars and fails if the checked-in fixtures differ by a byte.
|
||
Forgery cannot survive a real re-derivation.
|
||
|
||
Without (2), "generated from upstream" would be documentation. Do not weaken
|
||
either half: together they are what stops this oracle from decaying back into the
|
||
hand-typed fixture it replaced.
|
||
|
||
### What layer 3 actually proves
|
||
|
||
Read `scenarios.ts` literally. Cross-engine comparison is **outcome parity** — it
|
||
only catches a divergence severe enough to fail the flow. It cannot see settle
|
||
latching, retap counts, or a 1px truncation difference. Where finer behavior
|
||
matters we assert it **engine-side** via `engineInvariants` over agent-device's
|
||
own `replay-timing.ndjson`.
|
||
|
||
Layer-3 flows live in `differential/flows/` and drive the **real fixture app**
|
||
(`examples/test-app`, `com.callstack.agentdevicelab`), which the workflow builds
|
||
and installs. They are deliberately **not** the layer-1 corpus: those flows exist
|
||
only to be *parsed* — they name a fictional `com.example.app` and elements that
|
||
exist on no device — so a device run against them would fail before exercising
|
||
any runtime behavior, making the settle detector silently vacuous. A test
|
||
enforces the separation, and the workflow hard-fails if the app is not installed.
|
||
|
||
### Declared divergences (`knownDivergence`)
|
||
|
||
Layer 3 has the same contract as layer 1: **every divergence is a decision on the
|
||
record.** When the differential catches a real engine bug, the instrument does not
|
||
block on repairing what it just measured — the scenario declares it with a
|
||
`knownDivergence: { reason, tracking }`, the scheduled run stays green on that
|
||
known gap, and only *undeclared* divergences fail.
|
||
|
||
Two rules keep that from rotting, both enforced by `run.test.ts` / the runner
|
||
rather than by good intentions:
|
||
|
||
- **`tracking` is required** and must be a real issue URL. A declared divergence
|
||
with nothing behind it is how "temporarily expected" silently becomes permanent.
|
||
- **A stale declaration fails.** If a declared-divergent scenario starts passing,
|
||
the run goes red until the declaration is removed — so the fix PR must delete
|
||
it, and the oracle then enforces that the gap stays closed. The differential is
|
||
the acceptance test for its own findings.
|
||
|
||
A green run still prints what it is *not* proving.
|
||
|
||
This matters most for **bug class 4** (settle ordering), whose 200ms × 10 loop has
|
||
no reflectable upstream constant, so layer 3 is its only home. Its detector is the
|
||
invariant "a tap must not consume the entire settle budget" — a full-budget tap
|
||
(~2093ms against 2000ms) means the stability loop never latched, yet the flow
|
||
still passes, so outcome parity would miss it. The evaluator is pure and
|
||
unit-tested against synthetic traces (`invariants.test.ts`); only the device run
|
||
that produces a real trace is scheduled-only. A scenario that declares an
|
||
invariant fails if the trace is missing — a detector that cannot run is a
|
||
failure, not a pass.
|
||
|
||
## Files
|
||
|
||
- [`jvm-harness/`](./jvm-harness) — Gradle/Kotlin generator for layers 1–2. Depends on the
|
||
published Maestro jars; reads the parser and constants directly (never
|
||
transcribed). Requires JDK 17+.
|
||
- [`corpus/`](./corpus) — flows driven through the upstream parser.
|
||
`manifest.json` (generated by [`build-manifest.mjs`](./build-manifest.mjs), do
|
||
not hand-edit) records provenance: upstream flows are vendored verbatim with
|
||
their `sha256`; authored flows fill coverage gaps (`authored/`), encode the bug
|
||
classes (`bug-classes/`), and give the never-accept-what-upstream-rejects guard
|
||
teeth (`invalid/`). To add a flow: drop the `.yaml` in, add a note to `NOTES`
|
||
in `build-manifest.mjs`, and regenerate.
|
||
- [`fixtures/`](./fixtures) — the generated, checked-in layer-1/layer-2 captures.
|
||
- `packages/maestro/src/internal/conformance-normalize.ts` and
|
||
`conformance-selector-projection.ts` — package-private canonical projection and selector model.
|
||
- `packages/maestro/test/conformance/` — the deterministic verifier, declared divergences,
|
||
package-private harness, shared fixture seal, and layer-3 differential scenarios. Keeping this
|
||
code under package tests prevents parser/canonicalization tooling from widening the production
|
||
facade; regeneration imports the same package-owned seal implementation.
|
||
- [`regenerate.mjs`](./regenerate.mjs) — SHA-verifies the jars, rebuilds the
|
||
corpus manifest and fixtures, and seals them.
|
||
|
||
## Verify (per-PR, no Java)
|
||
|
||
```sh
|
||
pnpm maestro:conformance
|
||
```
|
||
|
||
The verifier fails on any **undeclared** divergence: a flow our engine parses
|
||
that upstream rejects (a conformance regression), an undeclared parse mismatch,
|
||
or an undeclared `we-reject`. Every `we-reject` must list the unsupported
|
||
command/option in `packages/maestro/test/conformance/expected-divergence.ts` — that
|
||
list is the mechanical parity record. A focused issue is attached only when
|
||
implementation work is planned.
|
||
|
||
## Regenerate (on an upstream-pin bump or intentional corpus/harness change)
|
||
|
||
Heavy, manual, needs JDK 17+ (Gradle is provided by the committed wrapper):
|
||
|
||
```sh
|
||
pnpm maestro:conformance:regenerate
|
||
```
|
||
|
||
This resolves the pinned jars, **verifies their SHA-256 against
|
||
`pinned-upstream.json`**, runs the harness over the corpus, and rewrites
|
||
`fixtures/`. Review the diff, then run the verifier. A pin bump is not required
|
||
when a corpus flow or harness behavior intentionally changes; regenerate from
|
||
the unchanged pin and review the generated diff. To bump the pin: update
|
||
`pinned-upstream.json` (version/tag/commit + the jar SHA-256s from Maven Central),
|
||
refresh the vendored corpus flows and their `manifest.json` `sha256`s, regenerate,
|
||
and reconcile any new divergences.
|
||
|
||
## Layer 3 (device)
|
||
|
||
```sh
|
||
pnpm maestro:conformance:differential -- --platform ios --out-dir .tmp/diff
|
||
```
|
||
|
||
Runs scheduled on the `conformance-differential` workflow, and on demand via
|
||
`workflow_dispatch`. `--dry-run` validates the scenario registry without a device.
|
||
|
||
The workflow builds and installs the fixture app, verifies its bundle id, pins
|
||
the Maestro CLI to the same version as layers 1-2, and passes `--maestro` so the
|
||
flow routes through the compat engine.
|
||
|
||
The built `.app` is **cached** (keyed on the app's sources, its dependency graph,
|
||
the iOS runtime, and the Xcode version), because building it costs ~22 minutes
|
||
versus ~6 minutes for the differential itself — 79% of the job, for an app that
|
||
changes almost never. A cache hit installs the bundle directly; a miss falls back
|
||
to the full build and repopulates. If you change anything under
|
||
`examples/test-app`, expect the next run to rebuild.
|
||
|
||
**Investigate locally, not through CI.** A device iteration in CI is ~40 minutes;
|
||
`--only` plus a local simulator is minutes:
|
||
|
||
```sh
|
||
pnpm test-app:install && pnpm --dir examples/test-app exec expo run:ios --configuration Release
|
||
pnpm maestro:conformance:differential -- --platform ios --only settle-after-tap --trace-root .agent-device
|
||
```
|