Michał Pierzchała
eb3fc5b28d
`ignorePatterns: ["packages/**"]` landed in #1494 W0 with the recorded reason "its resolver cannot follow workspace specifiers". That was either wrong at the time or never re-checked: the fallow version has not moved (^2.95.0 then and now) and it resolves @agent-device/* through each package's exports map today. packages/kernel alone exposes 8 subpaths and ~110 exports reachable only via workspace specifiers, and scanning it reports zero findings — a resolver that could not follow the specifier would report all of them. The cost of the ignore is that every package extraction silently removes its code from dead-code analysis. #1589 moved the selector engine into packages/selectors/ and shipped a façade with 15 zero-consumer exports, including `selectorUsesKey`, written in that PR and never called. A follow-up commit removed them by hand; nothing would have caught them. Removing the pattern surfaced 43 findings, driven to zero by deleting the dead code rather than by baselining or excluding it (fallow-baselines/*.json are empty on purpose — the posture is fix-or-document-the-exemption, so a first baseline entry would be a policy change): - 38 are deleted. 24 façade type re-exports whose only claim was that a consumer might one day want to name them — typecheck is green without every one, so the claim was theoretical; 5 façade value re-exports; 9 `export` keywords on symbols used only inside their own file. Every deleted façade symbol comes off scripts/layering/facade-symbols.ts (and ad-replay's inline pin in package-boundaries.test.ts) in the same change, so R11 is narrowed with the façade, never weakened around it. - 4 stale suppressions in src/provider-limrun-runtime.ts existed only because packages/ was invisible. - 5 have consumers analysis genuinely cannot see, and get an `ignoreExports` entry naming the consumer per the existing `comment` convention: four test-tree importers that --production does not walk, and `LimrunIosCommandExecution`, which src/sdk/limrun.ts republishes as agent-device/limrun — its only importer compiles in a temp checkout, so no static edge reaches it. test/integration/limrun-public-types.test.ts is the standing proof that one is real API. Three doc comments named types their façade no longer exports and are corrected rather than left asserting something false — including #1555's claim in session-replay-target-verification.ts that the daemon imports `AdReplayVerifiedTargetGuard` directly. It does not; it reaches that shape through `AdReplayTargetClassification`/`AdReplayDispatchGuard`, which is why the name read as dead. `scripts/maestro-conformance/**` was ignored wholesale to cover its corpus data. Narrowed to `corpus/**`, which un-hides the tooling beside it and turned up one more file-local export (`buildManifest`); regenerate.mjs's importer of `fixtureContentHash` becomes visible, so that needs no exemption at all. scripts/check-affected/model.ts deliberately did not select the `fallow` check for packages/*/src/**, carrying the same stale rationale as a comment. Without that selection the new scope would never run in the affected-driven lane, so the ignore removal would have bought nothing. model.test.ts now pins the selection. Verified: check:fallow and check:production-exports green with packages in scope; full-repo `fallow dead-code` back to its one pre-existing finding; typecheck, layering (R11), lint, format, build, check:package, and the limrun published-types integration test all pass. Probed by adding a fresh zero-consumer export to the xml façade — check:production-exports reports it, so the #1589 case now fails the gate. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Maestro conformance oracle
A three-layer oracle that proves the private agent-device Maestro package (packages/maestro)
stays faithful to a version-pinned upstream Maestro. It replaces the original
hand-typed parser fixture, whose transcribed expectations let four bug classes
slip through during #1217. Every expected value here is generated from the
pinned upstream artifacts — hand transcription is the failure mode this
replaces (issue #1274).
Pinned upstream: dev.mobile:*:2.5.1 (v2.5.1 / a4c7c95f), see
pinned-upstream.json.
Layers
| Layer | What it proves | Generated by | Runs in |
|---|---|---|---|
| 1 — parser | Our parser accepts/rejects/normalizes each flow like upstream | maestro-orchestra's YamlCommandReader over the corpus |
node --test, per-PR |
| 2 — semantics | Our geometry/retry/timing constants match upstream | ASM-read bytecode constants + parser-observed model defaults | node --test, per-PR |
| 3 — differential | Outcome parity on a device, plus engine-side timing invariants | Real Maestro vs agent-device test |
dispatch-only (see below) |
Layers 1–2 are checked-in generated fixtures (fixtures/) verified
deterministically with no Java. Layer 3 needs a device and the maestro CLI.
How "generated from upstream" is enforced
Per-PR CI cannot re-derive the fixtures — that needs Java, and staying Java-free on the normal path is a design constraint. So enforcement is two-layered:
- Per-PR: every fixture carries a
contentHashseal over its generated content, recomputed on each verify run. Editing a captured command or constant by hand breaks the seal. This is tamper-evident — it makes casual or accidental hand-editing impossible, but a determined editor could recompute it. - Scheduled (
conformance-regenerate): actually re-runs the JVM harness against the pinned jars and fails if the checked-in fixtures differ by a byte. Forgery cannot survive a real re-derivation.
Without (2), "generated from upstream" would be documentation. Do not weaken either half: together they are what stops this oracle from decaying back into the hand-typed fixture it replaced.
What layer 3 actually proves
Read scenarios.ts literally. Cross-engine comparison is outcome parity — it
only catches a divergence severe enough to fail the flow. It cannot see settle
latching, retap counts, or a 1px truncation difference. Where finer behavior
matters we assert it engine-side via engineInvariants over agent-device's
own replay-timing.ndjson.
Layer-3 flows live in differential/flows/ and drive the real fixture app
(examples/test-app, com.callstack.agentdevicelab), which the workflow builds
and installs. They are deliberately not the layer-1 corpus: those flows exist
only to be parsed — they name a fictional com.example.app and elements that
exist on no device — so a device run against them would fail before exercising
any runtime behavior, making the settle detector silently vacuous. A test
enforces the separation, and the workflow hard-fails if the app is not installed.
Declared divergences (knownDivergence)
Layer 3 has the same contract as layer 1: every divergence is a decision on the
record. When the differential catches a real engine bug, the instrument does not
block on repairing what it just measured — the scenario declares it with a
knownDivergence: { reason, tracking }, the scheduled run stays green on that
known gap, and only undeclared divergences fail.
Two rules keep that from rotting, both enforced by run.test.ts / the runner
rather than by good intentions:
trackingis required and must be a real issue URL. A declared divergence with nothing behind it is how "temporarily expected" silently becomes permanent.- A stale declaration fails. If a declared-divergent scenario starts passing, the run goes red until the declaration is removed — so the fix PR must delete it, and the oracle then enforces that the gap stays closed. The differential is the acceptance test for its own findings.
A green run still prints what it is not proving.
This matters most for bug class 4 (settle ordering), whose 200ms × 10 loop has
no reflectable upstream constant, so layer 3 is its only home. Its detector is the
invariant "a tap must not consume the entire settle budget" — a full-budget tap
(~2093ms against 2000ms) means the stability loop never latched, yet the flow
still passes, so outcome parity would miss it. The evaluator is pure and
unit-tested against synthetic traces (invariants.test.ts); only the device run
that produces a real trace is scheduled-only. A scenario that declares an
invariant fails if the trace is missing — a detector that cannot run is a
failure, not a pass.
Files
jvm-harness/— Gradle/Kotlin generator for layers 1–2. Depends on the published Maestro jars; reads the parser and constants directly (never transcribed). Requires JDK 17+.corpus/— flows driven through the upstream parser.manifest.json(generated bybuild-manifest.mjs, do not hand-edit) records provenance: upstream flows are vendored verbatim with theirsha256; authored flows fill coverage gaps (authored/), encode the bug classes (bug-classes/), and give the never-accept-what-upstream-rejects guard teeth (invalid/). To add a flow: drop the.yamlin, add a note toNOTESinbuild-manifest.mjs, and regenerate.fixtures/— the generated, checked-in layer-1/layer-2 captures.packages/maestro/src/internal/conformance-normalize.ts— package-private canonical projection.packages/maestro/test/conformance/— the deterministic verifier, declared divergences, package-private harness, shared fixture seal, and layer-3 differential scenarios. Keeping this code under package tests prevents parser/canonicalization tooling from widening the production facade; regeneration imports the same package-owned seal implementation.regenerate.mjs— SHA-verifies the jars, rebuilds the corpus manifest and fixtures, and seals them.
Verify (per-PR, no Java)
pnpm maestro:conformance
The verifier fails on any undeclared divergence: a flow our engine parses
that upstream rejects (a conformance regression), an undeclared parse mismatch,
or an undeclared we-reject. Every we-reject must list the unsupported
command/option in packages/maestro/test/conformance/expected-divergence.ts — that
list is the mechanical parity record. A focused issue is attached only when
implementation work is planned.
Regenerate (only on an upstream-pin bump)
Heavy, manual, needs JDK 17+ (Gradle is provided by the committed wrapper):
pnpm maestro:conformance:regenerate
This resolves the pinned jars, verifies their SHA-256 against
pinned-upstream.json, runs the harness over the corpus, and rewrites
fixtures/. Review the diff, then run the verifier. To bump the pin: update
pinned-upstream.json (version/tag/commit + the jar SHA-256s from Maven Central),
refresh the vendored corpus flows and their manifest.json sha256s, regenerate,
and reconcile any new divergences.
Layer 3 (device)
pnpm maestro:conformance:differential -- --platform ios --out-dir .tmp/diff
Runs scheduled on the conformance-differential workflow, and on demand via
workflow_dispatch. --dry-run validates the scenario registry without a device.
The workflow builds and installs the fixture app, verifies its bundle id, pins
the Maestro CLI to the same version as layers 1-2, and passes --maestro so the
flow routes through the compat engine.
The built .app is cached (keyed on the app's sources, its dependency graph,
the iOS runtime, and the Xcode version), because building it costs ~22 minutes
versus ~6 minutes for the differential itself — 79% of the job, for an app that
changes almost never. A cache hit installs the bundle directly; a miss falls back
to the full build and repopulates. If you change anything under
examples/test-app, expect the next run to rebuild.
Investigate locally, not through CI. A device iteration in CI is ~40 minutes;
--only plus a local simulator is minutes:
pnpm test-app:install && pnpm --dir examples/test-app exec expo run:ios --configuration Release
pnpm maestro:conformance:differential -- --platform ios --only settle-after-tap --trace-root .agent-device