mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
v0.20.3
38 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 devin-ai-integration[bot]
|
885c1486bb |
test(ci): single-retry policy for enumerated contention-flaky files (timeouts only) (#1448)
* test(ci): single-retry policy for enumerated contention-flaky files * fix: satisfy fallow * test(ci): read failures through a lane reporter so timeouts stay distinguishable * test(ci): cover the lane reporter and drop its duplicated boilerplate * chore(fallow): own the retry lane's tool-loaded export seams * test(ci): block retries on non-test failures and classify timeouts structurally * test(ci): decide retry eligibility from runner metadata and route gate verdicts through blockers * refactor(ci): name the retry policy's rules in code instead of comments * test(ci): mark runner-aborted timeouts inside the runner instead of inferring them * test(ci): make timeout provenance a per-run secret, not a writable flag Cover direct task.meta mutation in the real child-Vitest fixture gate. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ci): retry the failed files in the first run's project and coverage modes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: drop deleted repo-health file from the retry list after #1480 Rebase onto main post-#1480: the SkillGym/repo-health descope deleted scripts/repo-health/run.test.ts, whose CONTENTION_RETRY_FILES entry would now fail this PR's own missing-file check, and inlined the slow-test budgets into the reporter, resolving the budgets-module import. Envelope comments now point at scripts/lib/lane-envelope.ts instead of the closed #1430. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
53e4be5f86 |
Remove SkillGym suite and repo-health snapshot infrastructure (#1480)
* chore: drop SkillGym and the repo-health aggregator (#1412 descope) Remove the SkillGym harness (test/skillgym/), its check-affected lane, package scripts, and devDependency — the help-conformance bench is now the single non-gating small-model oracle. skills/ markdown classifies as docs in the affected-check selector instead of failing open. Remove scripts/repo-health: its only gating assertion duplicated the Layering Guard job, its case-count metric imported the deleted SkillGym suite, and its sole planned consumer (#1424 / PR #1477) was closed with the Track C descope on #1412. Verified: check-affected node --test suites, oxfmt, oxlint, tsc, check:layering, fallow audit vs origin/main, and the full unit suite (unit-core + subprocess-stub) all pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep * fix(scripts): fold slow-test budgets into the reporter for production-exports The Fallow production-exports gate flagged all three budget exports: their in-file consumer (SLOW_TEST_RATCHET) and the repo-health entry point that kept the module reachable were both removed in the descope, leaving the config-loaded reporter as the only consumer — invisible to --production analysis. The data-only module's second consumer is gone, so per the boundaries-are-earned norm the constants move into the reporter instead of gaining a suppression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep * docs: align skills/ format policy and purge last SkillGym mention Address both P2 review findings on #1480: the testing-matrix row and the selector's formatGate both still claimed oxfmt covers skills/, while selectChecks classifies skills/*.md docs-only (oxfmt ignores **/*.md, so the claim was a no-op even before). The matrix now states the docs-only policy and formatGate drops the dead underSkills fact. The merged examples/README.md index (from #1469) loses its skillgym mention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
630dc7c99b |
feat(examples): add runnable Node.js SDK examples under examples/sdk/ (#1469)
* feat(examples): add runnable Node.js SDK examples under examples/sdk/ examples/test-app is a fixture and the repo's only prior examples/ content; the real SDK usage patterns lived only in website/docs/docs/client-api.md with no runnable script anywhere. Adds four standalone, typechecked examples covering the minimum surface from #1463: root client session (create -> open -> snapshot/tap -> close with typed error handling), agent-device/metro (normalizeBaseUrl/resolveRuntimeTransport), agent-device/contracts (centerOfRect on a snapshot node), and agent-device/batch (runBatch for a custom transport). Each imports the published `agent-device/...` subpaths rather than relative src/ paths. examples/sdk/tsconfig.json path-maps those subpaths to src/sdk/ so `pnpm typecheck` (now also run against this tsconfig) checks the examples in CI without a prior build, workspace link, or publish step. Running an example for real still resolves `agent-device` as a self-referencing package after `pnpm build`. src/__tests__/client-api-examples-drift.test.ts guards the examples against drifting from client-api.md's subpath API manifest in both directions, picked up automatically by the existing unit-core vitest project (no new script or workflow needed). examples/README.md indexes the new examples and notes that test-app/ remains a fixture, not an example; it is not renamed or moved. Refs #1463 * fix: address Fallow findings on the new SDK examples Fallow flagged the four examples/sdk/*.ts files as unused files (not reachable from any entry point) and three functions as high complexity. - Register the examples as manual entry points in .fallowrc.json, matching how other standalone scripts (scripts/patch-xcuitest-runner-icon.ts, scripts/runner-request-count/run.ts) are already declared. - Reduce complexity in client-session.ts and contracts-result.ts by extracting device-resolution/error-reporting and rect-assertion helpers out of main(). - Reduce complexity in the drift guard's parseSubpathManifest by splitting bullet-matching and backtick-name extraction into their own functions. Verified: pnpm check:fallow --base <PR base sha> now reports no issues, and pnpm check:tooling / pnpm test:unit stay green. Refs #1463 * fix: compile client-api.md's actual code snippets, not just its symbol manifest Addresses review feedback on #1463's drift guard: the existing guard only parsed the doc's "Public subpath API" bullet manifest and compared imported symbol names, so a fenced ```ts snippet could drift or stop compiling without the guard noticing. Added test/integration/client-api-doc-snippets.test.ts, which extracts every fenced ```ts block from client-api.md and typechecks it against the real agent-device/* sources (reusing examples/sdk/tsconfig.json's existing paths mapping, read via `tsc --showConfig` so there's one source of truth). Free identifiers that continue a `client`/`snapshot` from an earlier snippet are stubbed — typed against the real SDK return type, not `any`, so continuation snippets still get real checking. Lives in the Node integration lane (test/integration/*.test.ts), not vitest's unit-core: it spawns a real tsc Program, well past the unit suite's 2.5s budget. Running this check against the existing doc surfaced real, pre-existing snippet bugs (unrelated to the new examples), fixed here: - "sessions.artifacts": `result.cloudArtifacts` accessed without narrowing the `CloudArtifactsResult | DaemonArtifactsResult` union first. - "Device cloud sessions": `platform`/`device` were passed into the client constructor config, which doesn't accept them; moved to the `apps.open()` call where those fields actually belong. - "Android ADB providers": the inline `exec` handler had no parameter types, so it failed under strict/noImplicitAny; annotated with the real `AndroidAdbExecutorOptions` type. Two further gaps the check surfaced are pre-existing product/API-surface questions out of scope for this PR (not the new examples), so they're allowlisted in KNOWN_DOC_GAPS with comments rather than silently patched: - "Remote Metro helpers" documents prepareRemoteMetro/reloadRemoteMetro/ stopMetroTunnel/resolveRemoteConfigProfile as public, but none of them are exported from agent-device/metro or agent-device/remote-config today. - "Web sessions"/audio probe pass `platform` to `observability.network()`/ `.audio()`, but NetworkOptions/AudioOptions have no `platform` field even though the CLI's network/audio commands accept `--platform`. Refs #1463 * fix: close the doc-snippet compiler's stubbing hole and the two suppressed gaps Addresses the second round of review feedback on #1463's drift guard: 1. stubFreeNamesAndRecompile auto-stubbed every "Cannot find name" as `any`, so a typo like `cliet.apps.open()` would silently pass on the second compile. It now only stubs identifiers in an explicit allowlist (KNOWN_FREE_NAME_STUB_TYPES) — the real SDK-derived continuations (`client`, `androidClient`, `snapshot`) plus the doc's own invented host-glue names, each typed precisely rather than loosely. Anything else is left as a real compile failure. Added a regression test that feeds a `cliet` typo through the guard and asserts it fails. 2. KNOWN_DOC_GAPS filtered six real compiler errors out of the final assertion while the test claimed every snippet compiles. Investigated both and fixed the actual contracts instead of suppressing them: - `prepareMetroRuntime`/`reloadMetro` (src/metro/client-metro.ts) and `stopMetroTunnel` (src/metro/metro.ts) already existed and matched the doc's described workflow almost exactly (same result shape) but were never re-exported from `agent-device/metro`; same for `resolveRemoteConfigProfile` and `agent-device/remote-config`. Added the four exports and fixed the doc's stale function names (`prepareRemoteMetro`/`reloadRemoteMetro`) and one stale field name (`profileKey` -> `companionProfileKey` on the prepare call) to match. - `NetworkOptions`/`AudioOptions` (src/contracts/client-observability.ts) had no `platform` field even though the CLI's `network`/`audio` commands accept `--platform` for the same use case, and the client methods already forward the options object to the daemon generically (`executeCommand('network'|'audio', options)`) — so this was a type gap, not a runtime one. Switched both from AgentDeviceRequestOverrides to DeviceCommandBaseOptions (matching PerfOptions' existing pattern), closing the gap for real instead of stripping `platform` from the doc. - The "Android installFromSource()" snippet was missing its `createAgentDeviceClient` import outright; added it. KNOWN_DOC_GAPS is gone — every fenced snippet now compiles for real, and the test's assertion matches what it claims. 3. Switched the raw `execFileSync` calls to `runCmdSync` from src/utils/exec.ts, per AGENTS.md's process-execution invariant (this is a .ts integration test, not a packaging fixture that needs to stay dependency-free). Refs #1463 * docs: fix stale reloadRemoteMetro() prose reference to reloadMetro() The prose right after the Remote Metro helpers snippet still named the old function; the compile guard only checks the fenced snippet, not surrounding prose, so it didn't catch this leftover from the prior rename. Refs #1463 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
6f476b54b7 |
refactor(daemon): table-ify handleSessionCommands (#1458) (#1467)
Replace the 18-branch if-chain in handleSessionCommands with a handler table keyed by session-routed command name, satisfies-checked against DescriptorSessionRouteCommandName (derived from command-descriptor registry entries with route: 'session'). A missing handler entry is now a compile error instead of a runtime routing gap caught only by expectHandlerResponse. Command-kind groupings (inventory/state/ observability/replay) map several command names to the same shared handler reference, preserving the former getSessionCommandKind clustering without the runtime lookup. Deletes the fallow-ignore-next-line complexity waiver the if-chain required. getSessionCommandKind itself is no longer called from production code (only exercised by its own unit test), so it's added to the fallow production-exports allowlist alongside the existing test-only-consumer entries. Claude-Session: https://claude.ai/code/session_01U7JRr87E2QguHF36XrVbph Co-authored-by: Claude <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
edca35d122 |
chore(deps): Renovate config, packageManager-derived pnpm in CI, repo-wide format (#1444)
* chore(deps): add Renovate config and enforce packageManager pnpm version in CI Refs #1422 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: bump pnpm to 11.17.0 and format the whole repo with oxfmt format/format:check drop their hand-maintained path list: oxfmt already skips node_modules and honors .gitignore, so the only exclusion list is .oxfmtrc.json ignorePatterns. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mutation): accept either quote style in the affected-lane path filter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(deps): keep fixture-app runtime deps as individual Renovate PRs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8cce0ef6b8 |
test: ratchet mutation score over enumerated decision kernels (#1441)
* test: ratchet mutation score over enumerated decision kernels Adds a Stryker (vitest runner) mutation lane scoped to the decision kernels, a per-module baseline with tool/config provenance, and a ratchet that only lets scores rise. Non-gating until two consecutive stable weekly sweeps. Refs #1415 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: declare the mutation test-scope seam for production-export analysis Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: own kernel tests in the mutation registry and ship the #1430 lane envelope - restore bench:help-conformance, broken by a formatting-path edit - kernel test files select their module on PRs (registry `tests` + workflow paths), asserted to reach the kernel through the import graph - every mutation run writes the standard scheduled-lane artifact envelope - move src/utils/__tests__/errors.test.ts beside its source per the mirror rule Refs #1415, #1430 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: derive kernel test ownership and land the scheduled-lane health monitor Ownership of a kernel's tests is now computed from the static import graph (scripts/mutation/ownership.ts) instead of a hand-listed set, so a test that reaches a kernel indirectly -- src/__tests__/daemon-error.test.ts through src/daemon.ts -- selects that kernel on a PR. The PR lane triggers on every src test and shards the derived modules, keeping wall clock at one module. The lane envelope (#1430) is now written on every exit path with the stage it reached, so a crash before any mutant runs is distinguishable from a lane that never ran. Adds the derived cadence monitor (scripts/lane-health, daily workflow): scheduled lanes are enumerated from .github/workflows/ and reported dark, failing, or never-run against their own cron cadence. * fix: merge only Stryker reports from a shard directory The shard artifacts now carry the lane envelope beside mutation.json, and the merge globbed every .json under the download path, so the ratchet job fed the envelope to the report parser and died after the mutants had already run. * fix(mutation): fail on incomplete shard sets and envelope pre-run failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mutation): downgrade a passing envelope when a later lane step fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mutation): shard by registry, defer the PR lane, drop the bundled watcher Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: describe registry sharding and the deferred PR mutation lane Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mutation): make the pre-graduation tooling exception select real mutants Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mutation): give the worktree fixture commits their own identity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Michał Pierzchała
|
2d1d70613f |
feat(bench): renderer-pinned samples, topic-coverage gate, error-recovery quizzes; trim skillgym to agentic checks (#1411)
* feat(bench): renderer-pinned samples, topic-coverage gate, error quizzes; trim skillgym to agentic checks The help conformance bench's quoted CLI output is now sourced from scripts/help-conformance-sample-outputs.mjs, and every sample is rebuilt through the real production renderers (settle output formatters, printHumanError, formatSnapshotText, refMutationAdmissionResponse) by scripts/__tests__/help-conformance-sample-outputs.test.ts — a rendering or message change fails deterministically instead of leaving the bench grading against output the CLI no longer prints. This retires the fabricated recoverable-failure envelope (production never throws a textual settle timeout; that case is replaced by a real DEVICE_IN_USE recovery quiz). Bench cases move to scripts/help-conformance-cases.mjs and are enumerated against the help-topic registry: helpTopicIds() is exported from cli-help, and scripts/__tests__/help-conformance-topic-coverage.test.ts fails when a help topic has neither a bench case nor an explicit waiver. New case families: error-envelope recovery quizzes (device-in-use, stale pinned ref, ambiguous find match, app-not-installed) pinned to real error text, topic coverage for tv/web/react-native/debugging/workflow, and a metamorphic twin of the settled-diff quiz. The skillgym smoke suite shrinks from 119 cases to the 5 that measure what only an agentic runner can show: skill routing plus output interpretation with a proven local CLI help probe (local-cli-help-policy). Its embedded samples now import the same pinned constants, replacing hand-transcribed output that had already drifted from the renderer. Knowledge checks belong to the bench; live fixture behavior belongs to the iOS simulator e2e suite. * review: drive error samples through the real producers; enforce local-help on the routing smoke The DEVICE_IN_USE, AMBIGUOUS_MATCH, and APP_NOT_INSTALLED parity tests no longer hand-author the producer message before rendering: each drives the actual producer — buildDeviceInUseBySessionError (extracted in session-open.ts and called by the handler), buildAmbiguousMatchError (now exported from find.ts), and buildAppNotInstalledError (extracted in app-resolution.ts and thrown by the resolver). Because each factory is exported from its producer file and called by the production path, dropping the production call would make it test-only and fail check:production-exports — the wiring is gate-enforced, not conventional. open-and-snapshot now sets requireLocalCliHelp and allowOnlyLocalCliHelpCommands, so the 'skill plus local help' claim is observed rather than assumed; without them the case can pass on model prior alone. |
||
|
Michał Pierzchała
|
317469f35a |
chore: remove dead type re-exports, prune stale suppressions, gate unused-types (#1364)
* chore: remove dead type re-exports and gate unused-types Clears the 72 `unused-types` findings left by #1363 and flips `rules.unused-types` from `warn` to `error` so new ones cannot creep in. Every fallow detector is now at zero. 71 of the findings were re-export lines (`export type { X } from './y.ts'` with no importer of X through that path); one was a direct declaration (`HelpTopicName`, zero references anywhere). Removing them cascaded, which is the point: imports that existed only to feed a re-export became unused (8 of them, caught by `noUnusedLocals`), and clearing those exposed a further chain through `command-projection.ts` -> `batch/index.ts` -> `batch/projection.ts`. Also updated a comment in `batch-policy.ts` that described the `command-surface.ts` re-export path this removes, and dropped two `export type {}` husks left where every name in a block was dead. One finding was NOT what it looked like: fallow flagged `DebugSymbolsOptions` and `DebugSymbolsResult` in `apple/core/debug-symbols/types.ts`, but removing them broke `apple/core/debug-symbols.ts`, which re-exported them from there. Both ends were dead — every real consumer imports from `contracts/debug-symbols.ts` — so the fix was removing the intermediate re-export too, not restoring the leaf. Safety: the published type surface is unchanged. All 11 entry points in package.json#exports export exactly the same names before and after. Two chunk files differ only because `AndroidSnapshotBackendMetadata` relocated between internal code-splitting chunks, which is not a consumer-visible boundary. * chore: prune fallow suppressions that no longer suppress anything The whole point of this arc was that fallow looked clean mainly because of its own ignore list, so the list itself deserved an audit. Emptying `ignoreExports` and re-running shows most entries no longer match a real finding: 20 blocks collapse to 5, with no change in what either fallow invocation reports. Removed 8 fully-stale blocks. They went stale for three different reasons: - code moved (`assertSafeDerivedCleanup` is exported from runner-cache.ts, not the runner-xctestrun.ts the suppression named), - the exports became genuinely used (apps.ts, runner-contract.ts, runner-session.ts, cloud-webdriver.ts, the test-utils fixtures), - and `installAndroidInstallablePath` was suppressed in two places at once. `app-lifecycle.ts` needed a code fix rather than a suppression: it re-exported three `parseAndroid*` helpers from app-parsers.ts that nothing imports through it, so the re-export is dropped and the block goes away. The two type re-exports on that statement ARE consumed and stay. The seven `src/daemon/handlers/*` blocks are KEPT, collapsed into one documented glob. They are not stale: those handlers are reached only through the dynamic `import()` table in request-handler-chain.ts, which the --production analysis behind `check:production-exports` cannot follow. A first pass removed them because the staleness probe only exercised the default config; the repo runs fallow twice, and `check:production-exports` caught it. |
||
|
Michał Pierzchała
|
c0fc822e80 |
chore: remove dead code and tune fallow's dead-code rules (#1363)
Evaluated knip (webpro-nl/knip) against the fallow setup already in the
repo, cleaned up everything it surfaced, then removed knip again: measured
head-to-head on the same tree, fallow is a strict superset once two
switches it already supports are flipped.
Dead code removed:
- `daemon/artifact-materialization.ts` (224 lines) had no production
caller, only its own test. Removing it exposed that
`downloadArtifactToTempDir` and the whole URL-fetch-with-redirects path
in `artifact-download.ts` were reachable only through it — the live
upload paths use the incoming-request helpers instead. That file goes
348 -> 123 lines. `readZipEntries` then fell out of `artifact-archive.ts`.
- Dead test-helper exports: 12 unused re-exports and 6 needlessly-exported
mocks in `session-test-harness.ts`, dead barrel entries in
`__tests__/test-utils/index.ts`, plus `withMockedXcrun`, `matchesSchema`,
`IOS_FRAME`, `IOS_TAB_FRAME`, `snapshotWithOffscreenContent`.
- `androidSnapshotHelperOutput` was duplicated byte-for-byte in
`provider-scenarios/android-world.ts`; it now imports the shared copy.
- 8 unreferenced type aliases, and 17 redundant type re-export lines in
`client/client-types.ts`. The published `.d.ts` is byte-identical before
and after all 19 files: those types already reach consumers through
`contracts/*` via `CommandResult<...>`, so this is not an API change.
Tooling:
- `.fallowrc.json` gains `includeEntryExports`,
`ignoreExportsUsedInFile: {type, interface}` and `unused-types: warn`.
That combination is what made the findings above visible; the previous
config was quiet mainly because of its own suppression list.
- Dropped 2 now-obsolete `ignoreExports` suppressions, added 3 documented
ones (published `sdk/*` surface, tool-config `default` exports, and the
`AssertTrue<...>` totality guards that exist only to satisfy
`noUnusedLocals`).
`unused-types` stays at `warn`: 72 pre-existing type re-export lines across
27 files remain, tracked separately. Every other fallow detector is at zero.
|
||
|
Michał Pierzchała
|
856d5d4900 |
test: replace the hand-typed Maestro fixture with a generated conformance oracle (#1289)
* test: replace the hand-typed Maestro fixture with a generated conformance oracle Closes #1274. The old harness (scripts/maestro-conformance*) compared 5 hand-authored flows against a hand-typed transcription of Maestro 2.5.1's command model. It proved parser self-consistency, not conformance: all four bug classes that cost #1217 days of live debugging slipped past it by construction, and it verified no upstream SHAs despite parsing them. Every expected value here is generated from the pinned upstream artifacts. dev.mobile:maestro-orchestra:2.5.1 is published on Maven Central, so the harness runs the real parser and reads the real bytecode — no full Maestro source build. Layer 1 (parser): a Gradle/Kotlin harness drives the pinned YamlCommandReader over a corpus of 42 vendored maestro-test flows (sha256-recorded) plus authored bug-class, coverage, and invalid flows, capturing each parse. The verifier parses each flow with the live engine and classifies it identical / both-reject / we-reject / mismatch / we-are-lenient. Every non-identical outcome must be a declared divergence, so the 17 we-reject entries in expected-divergence.ts are the mechanical parity backlog (assertTrue, clipboard, travel, killApp, and option-level gaps) rather than silent drift. Layer 2 (semantics): ASM reads static-final constants straight from the pinned bytecode without initializing driver classes (MAX_RETRIES_ALLOWED=3, SCREENSHOT_DIFF_THRESHOLD=0.005, ANIMATION_TIMEOUT_MS=15000, erase cap, and the iOS pre-tap gate we intentionally omit), plus the parser-observed 400ms swipe default. Each is cross-checked against MAESTRO_COMPATIBILITY_PRESETS. Layer 3 (differential): scheduled device scenarios. Cross-engine comparison is outcome parity only and says so; finer behavior is asserted engine-side via invariants over replay-timing.ndjson. Bug class 4's detector — a tap must not consume the whole settle budget, since a full-budget tap means the stability loop never latched while the flow still passes — is pure and unit-tested against synthetic traces; only the device run is scheduled-only. regenerate.mjs verifies the pinned jar SHA-256s before trusting output and is byte-deterministic across runs. Layers 1-2 verify in normal CI via node --test with no Java (the job installs deps: unlike the layering guard it copies, the verifier parses with the live engine, which imports the `yaml` package). Acceptance: the four bug classes each have a fixture; every command in SUPPORTED_MAESTRO_COMMAND_NAMES (the parser's own dispatch table, now exported as the single source of truth) is corpus-covered or listed unverified; the five documented deviations are expected-divergence entries. * fix: address review findings on the conformance oracle P1 — layer-3 scenarios could never run. They pointed at layer-1 corpus flows, which exist only to be PARSED: they name a fictional com.example.app and elements that exist on no device. A device run would have failed before exercising any runtime behavior, making bug class 4's detector silently vacuous. Layer 3 now has its own flows under differential/flows/ driving the real fixture app (examples/test-app, com.callstack.agentdevicelab); the workflow builds and installs it and hard-fails if it is missing. A test enforces the separation so a scenario can never point back at the parse corpus. Nothing else in this repo builds or installs the Expo fixture app, so those steps are new and unproven. The workflow is therefore dispatch-only: the cron is removed until a supervised first run proves the path. A nightly job that fails at 05:00 every day teaches nothing. P2 — layer 3 installed whatever version the online installer served. It now pins MAESTRO_VERSION from pinned-upstream.json, so layer 3 cannot drift from the version layers 1-2 claim, and asserts `maestro --version` matches. P2 — fixture content was not bound to regeneration. CI compared only the embedded upstream metadata, so a hand edit to a captured command or constant passed: the transcription failure mode this oracle exists to remove. Two-layer fix, because per-PR CI must stay Java-free and cannot re-derive: - Each fixture now carries a contentHash seal that the verifier recomputes, so editing a capture breaks the build. Tamper-evident, and tested by actually tampering rather than assuming a hash comparison works. - New scheduled conformance-regenerate job re-runs the harness against the pinned jars and fails on any byte difference. Forgery cannot survive a real re-derivation. This is what makes "generated from upstream" enforced. P3 — boot-ios-test-simulator requires runtime-version; now passed alongside preferred-device-name, as the other iOS workflows do. * tmp: trigger layer-3 differential on this branch to prove the device path workflow_dispatch cannot run pre-merge (it registers from the default branch), so this temporary push trigger exists only to execute the never-run device path on the PR head and capture evidence. Removed before merge. * fix(ci): install the fixture app unfrozen for the layer-3 device run First live run of the device path failed at the very first step: ERR_PNPM_LOCKFILE_CONFIG_MISMATCH. CI implies --frozen-lockfile and the fixture app's lockfile is out of sync with its package.json overrides. No CI job has ever built examples/test-app, so that drift was never surfaced. * fix: drop --ignore-workspace from test-app:install (defeats #649 security overrides) The first live run of the layer-3 device path failed at ERR_PNPM_LOCKFILE_CONFIG_MISMATCH, and the cause is a real latent bug rather than a stale lockfile. #649 moved the fixture app's `overrides` into examples/test-app/pnpm-workspace.yaml precisely because pnpm only honors overrides from a workspace root — they pin transitive deps (ws, brace-expansion, xmldom, postcss, uuid, shell-quote) to versions that clear Dependabot alerts. But `test-app:install` passes --ignore-workspace, which ignores that very file, so the overrides are dropped and no longer match the lockfile that has them baked in. It goes unnoticed locally because interactive installs are not frozen, and no CI job has ever installed this app. Dropping --ignore-workspace makes examples/test-app resolve as its own workspace root (it has its own pnpm-workspace.yaml and is not a member of the repo-root workspace), so the overrides apply and a frozen install succeeds. Verified both directions locally: with the flag + --frozen-lockfile reproduces the CI failure; without it, a frozen install completes and the lockfile's overrides stay intact. Note the workaround this replaces would have been actively harmful: installing with --no-frozen-lockfile resolves the mismatch by regenerating the lockfile WITHOUT the overrides, silently reverting the app to the vulnerable transitive versions #649 pinned away. * fix: make layer-3 scenarios prove what they claim, and parse the Maestro version Run 3 (29497919702) got the whole device path working: Expo build (30m), app installed, simctl check, pinned Maestro CLI install. Only the version ASSERTION failed — `maestro --version` prints an analytics banner before the version, and `tr -d '[:space:]'` mashed banner+version into one string. The CLI was correctly 2.5.1. Match the semver line instead, and set MAESTRO_CLI_NO_ANALYTICS (CI should not phone home). Verified the parse against the exact CI output: banner and clean forms both yield 2.5.1, wrong/empty still fail. tap-retry-if-no-change was vacuous: it tapped a navigating control, so the first tap always succeeded and retryIfNoChange never ran — it passed while proving nothing. It now taps the app's non-interactive title so the screen cannot change and the retry path is forced, and asserts tapRetries >= 1 from the trace (MaestroRuntimeMetrics already records it per step). A new metricAtLeast invariant kind carries the assertion; a test reproduces the old vacuity. percent-swipe no longer claims bug class 1. Truncation vs rounding is a <=1px delta that no app-observable device outcome can distinguish, so pass/pass could never back that claim up. The runtime half is instead pinned exactly by a pure unit test of resolveMaestroCoordinate (it short-circuits on a known viewport, so no device is needed) — verified to catch the regression by flipping trunc->round, which turns 3 of 6 tests red. Truncation had no test coverage at all before this. A test now forbids any device scenario from re-claiming bug class 1. * fix(ci): pass --maestro and match the fixture app's real UI in layer-3 flows Run 4 (29500262301) reached the differential itself — build, install, simctl check and the pinned Maestro 2.5.1 verification all passed — and surfaced two real bugs, both mine: 1. The runner invoked `agent-device test <flow>` without --maestro, so every scenario failed with "test does not support this file type". The repo's own scripts/run-test-app-maestro-suite.mjs passes it; the flag is what routes a .yaml through the Maestro compat engine. 2. settle-after-tap and percent-swipe assumed home-open-form is on screen at launch. It is not: real Maestro reported "Element not found: home-open-form", and the app's own helper flow scrolls it into view first. settle-after-tap now scrolls before tapping, mirroring that helper; percent-swipe no longer navigates at all and swipes the scrollable home screen, so it tests the conversion and nothing else. The remaining two flows already reported maestro=pass, so only the agent-device invocation was wrong for those. Note the settle invariant correctly reported "no-data: no completed tapOn steps" and FAILED rather than passing — a detector that cannot run is a failure, as intended. * feat: declare layer-3 divergences and schedule the differential Layer 3 ran both engines for the first time (29504440599) and immediately found a real engine bug. Blocking the measurement instrument on repairing what it just measured inverts the dependency, so layer 3 now gets the contract layer 1 already had: every divergence is a decision on the record. Adds `knownDivergence: { reason, tracking }` to the scenario type — the layer-3 twin of FLOW_DIVERGENCES. A declared divergence keeps the run green; only UNDECLARED ones fail. Two rules stop that from rotting, both enforced mechanically rather than by prose discipline: - `tracking` is required and must be a real issue URL (run.test.ts), because a declaration with nothing behind it is how "temporarily expected" becomes permanent without anyone deciding to. - a stale declaration FAILS: if a declared-divergent scenario starts passing, the run goes red until the declaration is removed. The fix PR must delete it, and the differential then enforces the gap stays closed — the oracle is the acceptance test for its own findings. Declared: - settle-after-tap -> #1299. Our scrollUntilVisible times out finding home-open-form where Maestro 2.5.1 scrolls to it and passes. Real engine correctness bug in an advertised command, found by this differential. Blocks bug class 4's device detector until fixed. - tap-retry-if-no-change -> #1300. The invariant caught the scenario being vacuous: both engines pass but tapRetries was 0, so retryIfNoChange never ran. Needs an inert fixture control; a scenario defect, not an engine one. Proven green on both engines and enforced now: percent-swipe, optional-warned-not-failed — the latter is real device-verified warned-vs-failed parity. With declarations in place the differential is green, so the schedule goes in (cron 05:00) per #1274. A green run still prints what it is not proving. * fix: park the flaky retry scenario instead of declaring it a divergence Run 29510020718 fired the stale-declaration guard on its first outing and caught my own mistake. tap-retry-if-no-change measured tapRetries=0 in run 29504440599 and tapRetries=1 in 29510020718 — same flow, same commit. So it is not vacuous as #1300 originally claimed: it is NON-DETERMINISTIC. The tap sometimes holds the hierarchy signature still and sometimes does not, because the fixture home screen carries live content. That exposes a real limit of the mechanism added in the previous commit: knownDivergence assumes the divergence REPRODUCES. A declared-but-flaky scenario flips between known-divergence (green) and stale-declaration (red) at random — a coin-flip scheduled job, which is worse than no scenario because it teaches people to ignore the differential. So the scenario is parked, not declared. The flow and the tapRetries invariant stay implemented and unit-tested, so the fix PR only re-adds the scenario once the fixture has an inert control. retryIfNoChange therefore has NO device coverage right now — tracked in #1300 and stated plainly rather than disguised by a green run. A test keeps it out of the active set until then. #1300 updated with the corrected diagnosis and both runs' evidence. Active differential: settle-after-tap (declared divergence, #1299), percent-swipe and optional-warned-not-failed (both enforced, pass/pass on real devices). * fix: make a knownDivergence waiver cover exactly one failure, not any failure P1 from re-review, and a real flaw: the code did not do what its own comment claimed. runScenario() collapsed every unexpected outcome and every invariant failure into `misbehaved`, then turned ANY of them green if the scenario carried a declaration. So while the #1299 scrollUntilVisible waiver is open, upstream Maestro could start failing too — or a different invariant could break — and the scheduled job would still report known-divergence and pass. A waiver for one bug was silently amnesty for the next. That is the exact failure this oracle exists to prevent, committed one commit after building the guard against it. knownDivergence now requires an `expected` signature: both engines' outcomes plus each declared invariant's status. The runner matches it exactly — - matches -> known-divergence (green, tracked) - misbehaves differently -> failed (red): not the failure the waiver covers - stops misbehaving -> stale-declaration (red): remove the declaration #1299's signature pins what runs 29504440599/29510020718 actually observed: maestro=pass, agent-device=fail, settle invariant no-data. Tests prove unrelated failures stay red under an open waiver: upstream also failing, our engine unexpectedly passing, a different invariant status, and a new invariant appearing are each NOT covered. A signature where both engines pass is rejected outright as describing no divergence. Also retains replay-timing.ndjson as a run artifact (review evidence note): the invariants are computed from that trace, so a report saying "tapRetries was 0" cannot be audited once the runner is gone without it. * perf(ci): cache the fixture app build for the layer-3 differential The differential job took ~30 minutes, of which 1331s (22 min, 79%) was building the Expo fixture app and only 347s was the differential itself — rebuilt from scratch on every run for an app that changes almost never. Cache the built .app, keyed on everything that can change the binary: the app's sources, native config, dependency graph, the build step itself, the iOS runtime, and the Xcode version. Mirrors the existing setup-apple-replay prebuilt-runner cache (same action pin, same Xcode-key + source-hash shape). On a hit the build is skipped entirely and the bundle is installed straight onto the booted simulator (~seconds), taking the job to roughly 8 minutes. On a miss it falls back to exactly the previous behaviour and repopulates, so the worst case is unchanged. The existing simctl verification still gates both paths, so a bad cache cannot produce a vacuous green: if the app is not installed, the job fails loudly rather than running scenarios against nothing. Note the first run after this lands is necessarily a miss. * refactor(ci): extract setup-fixture-app so any job can use the cached app The fixture-app build + cache was inline in the differential workflow, so nothing else could reach it. Extracted to a composite action mirroring setup-apple-replay, because the capability is what #320 has been missing: it wants replay coverage moved off Apple system apps onto a controlled fixture with stable ids, and that fixture (examples/test-app) already exists — CI just had no way to build and install it. The cache is genuinely shared. GitHub caches are per-repository and readable across workflows, and a run restores from its own branch or the default branch, so once a run on main populates it every workflow gets the hit and only the first one pays the ~22 minutes. The key is computed inside the action from a fixed input list and deliberately contains nothing caller-specific — folding a caller's workflow path into it would silently unshare the cache. Also removes a duplication risk: the action reads the bundle id from the built app's Info.plist rather than hardcoding it, so it cannot drift from what was actually built, and it fails loudly if the app is not installed. The conformance workflow keeps its own narrower assertion — that the installed id is the one its scenarios target — since that is its concern, not the action's. Usage: - uses: ./.github/actions/setup-fixture-app with: runtime-version: ${{ env.IOS_RUNTIME_VERSION }} # outputs: app-path, app-id, cache-hit * chore(ci): remove the temporary branch push trigger Run 29519848340 on this head executed both engines against the real fixture app and came back green, so the trigger that existed only to prove the never-run device path has done its job. Merged config is now cron (05:00) + workflow_dispatch, as required by #1274. known-divergence settle-after-tap maestro=pass agent-device=fail (#1299) ok percent-swipe maestro=pass agent-device=pass ok optional-warned-not-failed maestro=pass agent-device=pass This commit will not itself trigger a run: GitHub evaluates triggers at the pushed commit, and the push trigger is gone in it. |
||
|
Michał Pierzchała
|
e58cbcdb5f |
refactor: colocate native platform sources under android/, apple/, linux/ (#1273)
Move the scattered root-level native projects into per-platform folders and drop
the now-redundant platform prefix:
- android-ime-helper/ -> android/ime-helper/
- android-multitouch-helper/ -> android/multitouch-helper/
- android-snapshot-helper/ -> android/snapshot-helper/
- apple-runner/ -> apple/runner/
- macos-helper/ -> apple/macos-helper/
- src/platforms/linux/atspi-dump.py -> linux/atspi-dump.py
Only repo source paths move. Identity surfaces stay frozen so no user's runner
cache is invalidated on upgrade: the derived-cache key hashes source paths
relative to AgentDeviceRunner and excludes packageVersion, and the
~/.agent-device/{apple-runner,macos-helper} namespaces, the
agent-device-android-*-helper artifact/manifest/protocol names, the
AgentDeviceRunner Xcode project, and the `prepare ios-runner` CLI command are
unchanged. Updates build/package scripts, CI, package.json files+scripts,
ignore/attr/fallow configs, runtime path resolvers, and test fixtures.
Also: re-base repo-root-relative refs inside the moved apple/runner for the
added nesting level (gated XCUITest fixture walk + two doc links), and clean the
legacy dist/apple-runner packaged output so the relocated runner can't
double-ship into the wholesale-included dist (with a regression test).
|
||
|
Michał Pierzchała
|
fb1117f229 |
ci: ratchet against production-unused exports (#1202)
* ci: ratchet against test-only exports Three exported-and-unit-tested-but-unreferenced-in-production incidents this week (#1166 getNearestCommandNames, #1167 buildSettleTail, #1199 clearMetroSessionHints) — the first two were caught by fallow's dead-code check because they had zero importers anywhere; #1199 was missed because a test file imports the export, and fallow's default reachability graph counts a test import as "used". Adds a second, stricter pass reusing fallow's own --production mode (entry.exclude test/story/dev files) via scripts/test-only-exports/check.ts: an export alive in fallow's default graph but dead in its production graph, with no other reference anywhere in its own file, has no production call site — exactly the #1199 shape. Ratchets against a checked-in baseline (scripts/test-only-exports-baseline.json, 77 entries); new findings fail `pnpm check:test-only-exports` (wired into CI's Fallow job and check:tooling). A `// test-seam: <reason>` comment above an export is the escape hatch for intentional test seams. Also extends .fallowrc.json's ignoreExports for seven daemon route handlers (src/daemon/handlers/*.ts) that are genuinely production-reachable through request-handler-chain.ts's `typeof import()` lazy-load pattern, which fallow's static import graph can't trace as a named-export consumer — without this they were false positives in the production-mode pass. * fix: harden test-only-exports ratchet per review Addresses the two should-fixes and all five minors from the independent review of #1202: - Replace the regex own-file occurrence count with an oxc-parser AST walk (typescript@7 ships no JS scanner API, so the review's fallback tool suggestion is the primary): identifiers are counted as AST nodes deduped by source span, so mentions in JSDoc/block comments, strings, and template-literal text no longer masquerade as call sites (review finding 1, both constructed cases re-verified fixed), and a `//` inside a string no longer hides real usages (finding 6). Span dedupe keeps barrel re-exports (`export { x } from`) counting once. The sharper count surfaced one organic false negative on main: `selector` in src/commands/index.ts was previously exempted because the regex matched "selector" inside the './...selector-read.ts' import path string; it is now baselined alongside its sibling `ref` (same re-export line). - Make the baseline shrink-only (finding 2): --update-baseline refuses new findings with the same wire/delete/annotate message, so the `// test-seam:` annotation in the reviewed source diff is the only acceptance path; CONTRIBUTING no longer documents baseline regeneration as an acceptance option and now describes baseline growth as a deliberate manual edit. - Stale baseline entries now emit a `::warning` CI annotation (finding 3). - Commit a re-runnable fixture test (finding 4): check.test.ts mirrors scripts/layering/model.test.ts, builds a synthetic package with a clearMetroSessionHints-shaped export (JSDoc self-mention included), asserts it is flagged, and asserts the annotated twin passes; wired before the check in pnpm check:test-only-exports. - Mark the unreadable/unparseable-file fallbacks CONSERVATIVE: per CONTRIBUTING's convention (finding 5). - Document the dynamic property access (obj[name]) blind spot in the script header and CONTRIBUTING (finding 7). * fix: harden test-only export ratchet * refactor: use native Fallow export gate * chore: refresh production export baseline |
||
|
Michał Pierzchała
|
cd0cd16a9e |
build: migrate the library build from rslib to tsdown (Rolldown) (#1087)
* build: migrate the library build from rslib to tsdown (Rolldown) Replace the Rspack-based rslib build with tsdown, the Rolldown-based library bundler from the Vite toolchain family, so bundling, testing (Vitest/Vite), linting (oxlint), and formatting (oxfmt) all run on the same OXC/Rolldown stack. Outcome vs the rslib baseline (size-report): - build time: ~53s -> ~2s - JS raw +16.2 kB (+1.1%), JS gzip +2.7 kB (+0.6%) - the residual gap is OXC vs SWC minifier tightness, not chunking - npm tarball -3.0 kB - CLI --version startup ~3 ms faster; --help within the +/-5 ms measurement noise of interleaved A/B runs Chunk-merging experiments (single shared group, entries-aware groups, small-module groups) all regressed either total size or --help startup (a merged shared chunk adds +140 ms), so the default Rolldown split graph is kept. Custom codeSplitting groups also currently trip a rolldown-plugin-dts bug that re-emits type-only imports as runtime imports. Declarations still bundle per entry via tsgo; dist layout, entry names, and the internal/ worker/daemon entry resolution contract are unchanged. @microsoft/api-extractor was only consumed by rslib dts bundling and is removed together with @rslib/core. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqeW8sA2ZnvnftdvpCqFMS * ci: only cache the pnpm store when setup installs dependencies The layering-guard job uses setup-node-pnpm with install-deps: false, so it never creates a pnpm store. setup-node's post-job cache save then fails with a path validation error whenever the lockfile hash misses the cache - which any lockfile-changing PR does. Gate the cache on install-deps so no-install jobs skip pnpm store caching entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqeW8sA2ZnvnftdvpCqFMS --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
a3e967526a |
refactor: rename ios-runner -> apple-runner (#981) (#996)
Finish the cosmetic ios-runner -> apple-runner rename now that the top-level XCTest runner is the OS-agnostic Apple engine (iOS/iPadOS/tvOS/macOS/visionOS from one Xcode project). Cosmetic only, no behavior change: - git mv ios-runner/ -> apple-runner/ (AgentDeviceRunner, README, RUNNER_PROTOCOL) - Update repo project-path consumers: build-xcuitest-apple.sh, package.json files globs, .fallowrc.json, write-xcuitest-cache-metadata.mjs, runner-xctestrun.ts fingerprint/project paths, recording overlay + test, daemon-client-timeout kill pattern, setup-apple-replay hashFiles glob, ci.yml swift-compat scan, AGENTS.md. - Rename runtime home cache/derived/lease dir default ~/.agent-device/ios-runner -> ~/.agent-device/apple-runner (build script, package/clean scripts, runner-xctestrun RUNNER_DERIVED_ROOT, runner-lease, runner-contract hint, cli-help/commands.md docs) and the tests asserting it. - Rename OS-agnostic runner symbols: runIosRunnerCommand -> runAppleRunnerCommand, prewarmIosRunnerCache -> prewarmAppleRunnerCache, createIosRunnerCachePrewarmOnColdBoot / createIosRunnerCacheColdBootPrewarmForOpen -> createAppleRunner* (+ call sites, type aliases, test mocks). Intentionally left as ios-runner (out of scope / would change behavior): - prepare ios-runner CLI subcommand (user-facing command name) - AGENT_DEVICE_IOS_RUNNER_* env var names and .tmp/ios-runner-derived CI values - ios-runner-prebuilt cache-key-prefix, ci.yml job id, workflow/ADR filenames - agent-device-ios-runner-<version> release artifact basenames Part of #972 (Phase 3 - Apple PlatformPlugin). |
||
|
Michał Pierzchała
|
a707348b9b |
feat: add hosted WebDriver provider support (#948)
* feat: add cloud webdriver artifacts * fix: clean up local session after provider release failure * fix: tag cloud webdriver provider requests * feat: connect hosted webdriver providers * docs: document hosted provider credentials * refactor: tighten cloud webdriver provider internals * refactor: consolidate cloud webdriver provider definitions * refactor: collapse hosted webdriver runtime wrapper * refactor: reduce cloud webdriver smell surface * docs: clarify hosted provider interfaces * fix: avoid regex slash trimming in webdriver urls * docs: rename hosted providers to device clouds * fix: align provider profile imports with remote modules * fix: skip local android recovery for provider devices * test: classify cloud provider integration flags * fix: close active cloud connection session * test: cover provider disconnect cli flow * fix: make cloud webdriver sessions launchable * fix: align cloud webdriver input gestures * fix: avoid keyboard input during cloud scroll * fix: constrain cloud webdriver scroll gestures * refactor: isolate cloud webdriver scroll frame * refactor: deduplicate cloud webdriver helpers * refactor: tighten cloud webdriver action types * fix: harden cloud webdriver release * fix: polish provider disconnect diagnostics * refactor: group connection profile helpers * fix: repair rebased internal paths * fix: satisfy cloud webdriver CI guards |
||
|
Michał Pierzchała
|
87c43562d3 |
refactor: consolidate public SDK entry barrels into src/sdk/ — Phase 5 (#986)
Move the package's public entry points into a dedicated src/sdk/ folder so the
public surface lives in one clearly-owned place, per plans/perfect-shape.md §5.5
("sdk/ = re-export barrels only") and the §3 target DAG. Follows the #951/#960
pattern (behaviorless path codemod; rslib entry KEYS unchanged so dist output
paths — and therefore package.json `exports` — stay byte-identical).
Barrel structure (all 11 public subpaths now resolve through src/sdk/):
- 9 already-thin barrels moved verbatim (git rename, internal import paths
re-depthed by one ../): index, artifacts, metro, batch, remote-config,
install-source, android-adb, contracts, selectors.
- io and finders keep their implementation at src root (real logic + internal
importers); src/sdk/io.ts and src/sdk/finders.ts re-export them (export *).
package.json exports/main/types: UNCHANGED. Each rslib entry key is preserved
(index, io, ...), so dist output stays dist/src/<name>.{js,d.ts} and the
published subpaths + files do not move. No exports rewrite was needed.
Repoints:
- rslib.config.ts: 11 entry sources -> src/sdk/*.ts (keys unchanged).
- 7 public *-test imports -> src/sdk/*.
- 3 internal src importers of the selectors barrel (find.ts, selector-read.ts,
resolution.ts) split to import impl directly (utils/selectors-parse.ts +
daemon/selectors.ts) — removes internal dependence on the public barrel.
- .fallowrc.json entrypoints + vitest coverage exclude (src/sdk/** glob).
Published-surface verification (npm pack --dry-run before/after):
- exports map identical; 146 packed files identical EXCEPT one internal
code-split chunk renumbered (dist/src/9836.js -> 893.js, byte-identical
content).
- All 11 public .d.ts byte-identical; 10/11 public .js byte-identical;
selectors.js differs only by that internal chunk reference. Same symbols.
Gates: tsc 0 · rslib build 0 · oxfmt/oxlint clean · fallow audit (31 files) clean
· vitest unit 2883 pass.
|
||
|
Michał Pierzchała
|
3d70943550 |
feat: enforce import-direction DAG (Phase-5 layering lint) (#984)
Generalize the inline CI "Layering Guard" grep into a structured
import-direction lint (scripts/layering/check.ts) over the resolved
import graph, per plans/perfect-shape.md §5.5.
The full target DAG (kernel ◄ platforms ◄ core ◄ commands ◄ {cli,
client, daemon/server}; client ◄ daemon/client) is only partly realized
— the client/remote/metro extraction, the daemon/server split, and the
utils dissolution are still pending Phase-5 moves, so the tree still
holds legitimate back-edges (platforms→core, commands→cli, utils→*).
Enforcing the whole DAG today would need a mass import rewrite that
Phase 5 defers. The lint therefore enforces the three invariants the
completed moves (kernel/, daemon/client/) already guarantee and that are
green today:
R1 kernel-sink — nothing under src/kernel/ imports another zone,
except the one type-only kernel→contracts re-export.
R2 commands-floor — nothing below the command surface (kernel,
platforms, core, daemon) imports src/commands/.
Generalizes the former guard (daemon + platforms).
R3 platforms-seam — platforms/ is statically imported only at the
core interactor seam (src/core/interactors/) and by
the daemon server; elsewhere use a dynamic import()
or a type-only import, preserving CLI cold-start.
Dynamic import('../platforms/*') and `import type` stay allowed.
Fixes the three pre-existing R3 violations by converting static
platforms value imports to dynamic imports (all in already-async call
sites, behavior-preserving and cold-start-improving):
- src/client/client.ts debug.symbols → lazy symbolicateCrashArtifact
- src/cli/commands/web.ts setup/doctor → lazy agent-browser-tool
- src/core/dispatch-interactions.ts runner-sequence → lazy (matches the
file's own dynamic-import pattern)
Wire the check into the Layering Guard CI job and add a check:layering
package.json script (also folded into check:tooling). scripts/layering/**
is excluded from fallow (untested CI script, like scripts/perf/**).
|
||
|
Michał Pierzchała
|
3df1e145b4 |
docs: fix stale platforms/ios paths after the Apple consolidation (#968) (#969)
Post-#968 follow-up: the OS-agnostic Apple runner engine moved from
src/platforms/ios/ to src/platforms/apple/core/, but several docs/config still
pointed at the old locations, misleading agents that grep those paths.
- AGENTS.md: repoint the runner-seam map, the Apple-family sync rule, the
record/trace seam, the search-roots hint, and the platform-backends list at
src/platforms/apple/core/...; rename "iOS Runner Seams" -> "Apple Runner Seams".
- .fallowrc.json: drop the two stale ignoreExports entries for the removed
src/platforms/ios/apps.ts and src/platforms/ios/index.ts (the live
src/platforms/apple/core/apps.ts entry already covers those test-only exports).
- ios-runner/{README,RUNNER_PROTOCOL}.md: point the TypeScript-client links at
src/platforms/apple/core/runner/runner-client.ts.
Docs/config only; no behavior change. fallow audit + build/lint stay green, and a
live iOS simulator replay suite (6/6) confirms the consolidated runner works.
|
||
|
Michał Pierzchała
|
26ac865c63 | refactor: consolidate Apple platform internals (#968) | ||
|
Michał Pierzchała
|
edc8dd059b |
ci: automate iOS runner request-count gate for the Apple runner unwind (Phase 3 step c prep) (#966)
Replaces the manual "run with --debug, hand-count the runner phases" check with an automated, committed assertion so the Phase 3 step (c) runner relocation (and future runner refactors) can prove byte-identical runner request behavior. - src/daemon/runner-request-count.ts: pure, unit-testable counter. Parses the daemon --debug diagnostics ndjson and counts the iOS-runner round-trip phases, plus baseline parse/compare logic. Owns RUNNER_ROUND_TRIP_PHASES as the single source of truth, now imported by request-router.ts (was a local const) so the in-process cost graft and the external counter never drift. - src/daemon/__tests__/runner-request-count.test.ts: 13 unit tests over synthetic ndjson fixtures (tolerant parse, counting, baseline parse/compare). Run in the normal unit suite; no hardware. - scripts/runner-request-count/: assertion harness (run.ts) + committed baseline (expected-counts.json). Drives the existing smoke-ios replay scenario with --debug in an isolated --state-dir, counts runner round-trips from daemon.log, and asserts against the baseline. --update regenerates the baseline. Infra hiccups are inconclusive (don't fail); only a real count drift fails. - .github/workflows/ios.yml: new "Assert iOS runner request count" step in the smoke-ios job, reusing the booted simulator. - package.json: `validate:runner-count` script. .fallowrc.json: harness entry. The baseline ships unarmed (established=false); the harness records observed counts (printed + uploaded as a test/artifacts artifact) without failing, so the maintainer arms it once from a real CI run. |
||
|
Michał Pierzchała
|
1cbe446df5 |
chore: prune public package exports (#961)
* chore: prune public package exports * chore: drop extra facade exports * chore: remove dead export leftovers * docs: align public API docs |
||
|
Michał Pierzchała
|
189b062519 |
refactor: extract client/ folder — Phase 5 (#960)
Move the SDK client + companion-tunnel cluster out of the src/ root into a dedicated src/client/ folder, per plans/perfect-shape.md §5.5 (~8k LOC client/remote unfoldered at src root; remote/ already extracted in #951). Files moved (9): client, client-types, client-shared, client-normalizers, client-companion-tunnel{,-contract,-worker}, client-react-devtools-companion, companion-tunnel. - git renames; 68 importers repointed via the resolve-based codemod (intra-client stays ./, staying deps recomputed) - companion-tunnel.ts keeps rslib key 'internal/companion-tunnel' so dist output stays dist/src/internal/companion-tunnel.js (public subpath byte-identical; verified by build) - update non-src importers (3 integration tests, vitest coverage include), .fallowrc.json entrypoint, fallow-baselines/health.json keys backend.ts and daemon-client*.ts are intentionally left for follow-up (daemon/client split). Behaviorless path codemod; all gates green. |
||
|
Michał Pierzchała
|
548bf0c47c |
refactor: extract remote/ folder — Phase 5 (#951)
Move the remote/proxy/upload subsystem out of the src/ root cluster into a dedicated src/remote/ intent folder, per plans/perfect-shape.md §5.5: daemon-proxy · daemon-artifacts · upload-client(-artifact) · remote-config · remote-config-core · remote-config-schema · remote-connection-state - 8 files moved (git renames); imports repointed via a resolve-based codemod (path.relative recomputation — correctly distinguishes the root remote-config from the unrelated src/utils/remote-config.ts) - rslib entry keeps key 'remote-config' so dist output stays dist/src/remote-config.js; public 'agent-device/remote-config' byte-identical - update .fallowrc.json entrypoint + fallow-baselines/health.json keys + vitest.config.ts coverage include + the integration test import paths Behaviorless path codemod. typecheck/lint/build/fallow/tests all green. Stacked on #950 (contracts→kernel). |
||
|
Michał Pierzchała
|
b6a9bb2186 |
refactor: move contracts.ts into src/kernel — Phase 5 (#950)
Relocate the central contracts barrel into the kernel/ dependency sink alongside device/errors/redaction/snapshot (kernel now owns the pure domain types per plans/perfect-shape.md §5.5). - src/contracts.ts -> src/kernel/contracts.ts (git rename) - repoint all 44 internal importers to ../kernel/contracts.ts - rslib entry keeps key 'contracts' so dist output stays dist/src/contracts.js; the public 'agent-device/contracts' subpath is byte-identical (proven by the metro precedent in #947 and verified via build + package-exports test) - update .fallowrc.json entrypoint + fallow-baselines/health.json key Behaviorless path codemod (49 files, +57/-57). typecheck/lint/build/fallow audit/public-contract tests all green. |
||
|
Michał Pierzchała
|
3aa43bfc91 |
refactor: extract metro/ folder — Phase 5 (#947)
Move the metro cluster into src/metro/ per plans/perfect-shape.md §5.5 (`metro/ ← metro* · client-metro*`): src/metro.ts -> src/metro/metro.ts src/metro-types.ts -> src/metro/metro-types.ts src/client-metro.ts -> src/metro/client-metro.ts src/client-metro-companion.ts-> src/metro/client-metro-companion.ts Pure path codemod, no behavior change. Imports were rewritten by a resolve-based codemod (compares resolved absolute paths, not naive string match). Also updated the three non-.ts references to the moved paths: the rslib `metro` entry, the fallow entry list, and the fallow health-baseline key. The companion-tunnel cluster (client-companion-tunnel*, companion-tunnel, client-react-devtools-companion) stays at src/ root — it is a separate domain (the future `companion/` slice) and keeps the worker process entrypoint (companion-tunnel.ts / client-companion-tunnel-worker.ts) co-located with its spawner (client-companion-tunnel.ts), so the worker entry resolution is unchanged. `npm run build` emits both dist/src/metro.js and dist/src/internal/companion-tunnel.js. |
||
|
Michał Pierzchała
|
b8172e3da3 |
perf(daemon): offload PNG decode/encode and screenshot diff to a worker thread (#734)
* perf(daemon): offload PNG decode/encode and screenshot pixel diff to a worker thread PNG decode (inflateSync) and per-pixel screenshot diffing previously ran synchronously on the daemon event loop, stalling all concurrent sessions while multi-MB screenshots were processed. - add src/utils/png-worker.ts worker_threads entry (rslib internal/png-worker) handling one decode, encode, or diff-pixels job per message - add src/utils/png-worker-client.ts async wrappers (decodePngAsync, encodePngAsync, computeScreenshotDiffPixelsAsync) that lazily spawn the worker, resolve it next to the current module in dev (.ts) and dist (.js) like the companion tunnel entry, and fall back to the in-process synchronous path when the worker is unavailable - extract the unchanged pixel-compare loop into src/utils/screenshot-diff-pixels.ts so both paths share identical logic - route daemon call sites (screenshot-overlay annotate, compareScreenshots) through the async wrappers; results stay byte-identical https://claude.ai/code/session_01LXZXzxi55sZ11DSyqWyBA2 * refactor(daemon): apply review findings to PNG worker offload - guard postMessage failures: a job-specific send error (e.g. DataCloneError) now cleans up its pending entry and falls back to sync for that call only - resolve the worker entry via a shared src/utils/internal-entry.ts helper that returns null on any resolution failure (non-file import.meta.url) - match repo precedent: spawn the worker with --experimental-strip-types when the resolved entry is a .ts module - report permanent worker degradation once (scoped diagnostic + process warning with the failure reason) instead of silently going sync-only - daemon lifecycle: pre-warm the worker at startup and terminate it during shutdown with a 1s best-effort timeout (daemon-only) - collapse the three async wrappers onto one kind-typed job runner with a single unavailability channel (rejection, no null path) and drop the dead mismatched-result guards - derive the diff-pixels contract types from screenshot-diff-pixels.ts and share toBuffer via the contract module - serialize worker errors with normalizeError and reconstruct AppError (code/message/details) client-side; the worker reuses decodePng so decode failures are identical on both paths - transfer result buffers back to the client when a view fully owns its ArrayBuffer; clone pooled buffers to protect Node's shared buffer pool - decode baseline/current screenshots concurrently in compareScreenshots - move resizePngFileToMaxSize to src/utils/png-resize.ts and route its decode/encode through the worker (daemon screenshot --max-size path) https://claude.ai/code/session_01LXZXzxi55sZ11DSyqWyBA2 * fix(fallow): declare png worker entry and simplify resultTransferList The Fallow audit flagged src/utils/png-worker.ts as unreachable from any entry point and resultTransferList as over the complexity threshold. - Declare src/utils/png-worker.ts in .fallowrc.json's entry list: it is a worker_threads entry point loaded at runtime (mirrors src/daemon.ts / src/companion-tunnel.ts and the internal/png-worker rslib entry). - Flatten resultTransferList into filter/map over an extracted ownsEntireArrayBuffer predicate, preserving the exact transfer rule (byteOffset === 0, byteLength === owner.byteLength, real ArrayBuffer). - Add direct unit coverage asserting fully-owned buffers are transferred while pooled/offset views are not. https://claude.ai/code/session_01LXZXzxi55sZ11DSyqWyBA2 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Michał Pierzchała
|
45cfad5cc5 |
feat: e2e command perf benchmark harness + nightly CI (#630)
* feat: add e2e command perf benchmark harness + nightly CI Adds scripts/perf, a cheap end-to-end perf benchmark that drives the built CLI through an ordered Settings tour of ~24 commands for N rounds, on a fully isolated daemon/state-dir and self-cleaning device, and emits JSON + Markdown reports. Per-command timing comes from wrapping each batchable command in its own single-step batch (daemon durationMs) plus wall-clock around the process. Wires a scheduled + workflow_dispatch CI job (perf-nightly.yml) that reuses the cached iOS XCUITest runner (setup-apple-replay) and the Android replay host, and runs the CLI from source via --experimental-strip-types (no dist build). * refactor(perf): drive the harness CLI via runCmdSync, not spawnSync Review (P2): repo rule is to spawn processes through src/utils/exec.ts, not node:child_process directly. Switch the perf harness's invokeCli to runCmdSync (allowFailure so non-zero exits are recorded as samples) and add a maxBuffer option to ExecOptions/runCmdSync (snapshot payloads exceed Node's ~1MB default). * perf(harness): warm the runner after open so the first measured command is clean The first interaction after open/relaunch pays the one-time iOS XCUITest runner startup (~10s+ cold) and a per-relaunch first-AX-query settle cost (~4s). That was landing on the first measured command each round (snapshot -i), inflating it ~10x vs the next snapshot. Run an untimed warmup snapshot -i after establishSession, after each round's reset-open, and after every freshRoot relaunch, so no measured command absorbs runner startup. Noted in the report header. * refactor(perf): address review + fix Fallow CI - exec.ts: extract spawnRejectionError + commandCloseFailure helpers, deduping the error/close handler clones (Fallow duplication ✗ that surfaced once the maxBuffer change pulled exec.ts into the audit scope). - .fallowrc: exclude scripts/perf/** (non-shipped benchmark tooling, like examples/ test-app) so its naturally-moderate functions don't trip the complexity gate. - config.ts: drop unused exports CLI_BIN/DEFAULT_OUT_DIR; add readIntValue so --n/--rounds/--warmup report the actual flag + reject non-integers clearly. - harness.ts: extract toSample(); type sampleError param as CliResult. - scenario.ts: ScenarioStep is now a discriminated union on execMode (removes step.step!/ step.args ?? []). - comment/legend rewords (platform defaults are local-convenience/CI-overridden; elements = node count). check:fallow now green; typecheck/lint/unit pass. * perf(harness): downgrade sample ok when a batch step reports ok:false Defensive belt-and-suspenders for the Codex review note: stop-only batch already surfaces a failed step as a top-level failure (caught by invokeCli), but if an on-error=continue mode ever keeps the batch ok while a step fails, don't silently count that step as a successful sample — derive ok from the step's own result.ok. |
||
|
Michał Pierzchała
|
ab760b6cf9 | fix: apply app icon to iOS UI test runner (#611) | ||
|
Michał Pierzchała
|
094c290703 |
perf: speed up iOS replay runner (#557)
* perf: speed up iOS replay runner * fix: harden ios replay fast paths * fix: address ci validation failures * refactor: trim unused ios replay surface |
||
|
Michał Pierzchała
|
59d28e8446 |
refactor: add provider-first device lab tests (#542)
* refactor: add provider-first device lab tests * refactor: tighten device lab provider seams * test: cover provider lab contracts * docs: record device lab harness direction * ci: run device lab integration tests * test: move device lab under integration * test: extract device lab helpers * refactor: centralize apps filter defaults * test: drop lab-covered unit tests * test: fold platform happy paths into device lab * test: reuse device lab helpers * test: move device lab to in-process harness * test: replace session handler cases with device lab * test: harden device lab scenario contracts * docs: define unit test retention policy * test: expand provider device lab coverage * test: harden provider device lab coverage * test: cover manifest install and runner session contracts * chore: remove unused provider cleanup code * test: split android find device lab scenario * test: track provider lab architecture progress * test: clarify provider lab roadmap progress * test: advance provider lab session coverage * test: move menubar click routing to device lab * test: move menubar snapshots to device lab * refactor: centralize screenshot flag plumbing * refactor: colocate screenshot flag metadata * test: cover all public commands in device lab * test: move macos wait success to device lab * test: drop redundant perf and diff units * test: move push payload paths to device lab * test: move network parsing to device lab * test: move log cleanup to device lab * test: move log restart and boot to device lab * test: move ios physical boot to device lab * test: cover perf startup in device lab * test: extract android and ios device lab worlds * test: trim device lab world surface * test: split snapshot capture unit coverage * test: deepen device lab coverage and trim handler units * test: clean up device lab migration scaffolding * test: report device lab public command coverage * refactor: make Apple provider seams semantic * refactor: tighten device inventory and Linux provider seams * refactor: tighten request provider scoping * refactor: add semantic macos host provider * test: broaden device lab find coverage * test: cover workflow flags in device lab * refactor: promote linux input provider seam * test: clarify device lab flag coverage * test: classify snapshot force-full progress * test: enforce device lab progress in ci * test: stabilize device lab ci * test: move packaged metro smoke to integration * test: drop stale provider seam coverage * test: harden provider scope regression coverage * refactor: remove stale platform barrels * refactor: keep linux clipboard and screenshots semantic * refactor: move macos host tools behind provider * fix: honor remote artifact output paths * test: deepen runtime coverage for daemon and runner paths * test: share loopback test helpers * refactor: make daemon runtime importable * fix: honor replay target metadata * chore: tighten final device lab quality gates * test: share device lab setup helpers * test: remove generic apple lab fallback * test: deduplicate device lab helpers * chore: tighten fallow duplication signal * refactor: share apple diagnostic helpers * fix: detect active android ime during fill verification * test: consolidate provider-backed integration suite * ci: fix fallow and iOS smoke setup * chore: consolidate cleanup after ci fixes * test: split vitest unit and integration projects * docs: mention MCP discovery metadata * docs: add agent skills context pointers * fix: close provider recording coverage gaps * fix: restore mcp compatibility smoke * test: cover provider edge regressions * test: consolidate loopback helpers * docs: remove stale provider routing reference * fix: harden final provider review issues * chore: defer mcp cleanup from provider refactor |
||
|
Michał Pierzchała
|
c1550721ce |
fix: collapse Android snapshot row noise (#545)
* fix: collapse Android snapshot row noise * docs: clarify agent-facing snapshot text * fix: preserve semantics when collapsing Android snapshots * fix: trim redundant Android collapse hints * fix: collapse passive Android duplicate children * refactor: drop unused Android snapshot hint code * docs: clarify Android Metro verification guidance * test: cover Android Metro reverse planning * test: fix codex skillgym runner * fix: satisfy fallow snapshot checks * docs: clarify PR validation guidance |
||
|
Hwidong Bae
|
2e04edd869 |
fix: prevent runner XCTest attachment bloat (#520)
Long-running AgentDeviceRunnerUITests sessions can inherit Xcode 26 screenRecording/deleteOnSuccess defaults and leave hidden testmanagerd attachments behind when sessions terminate outside a clean XCTest success path. Pin an explicit AgentDeviceRunnerUITests.xctestplan with screenshots and keepNever attachment lifetimes, then normalize the generated per-session .xctestrun copy before test-without-building. The .xctestrun normalization protects stale cached artifacts and Xcode versions that ignore some test-plan attachment lifetime keys during build-for-testing. This only affects XCTest's automatic attachments; user-requested agent-device recordings still use the existing record command paths. Co-authored-by: Michał Pierzchała <thymikee@gmail.com> |
||
|
Michał Pierzchała
|
8aa4abe465 | feat: expand android adb provider boundary (#481) | ||
|
Michał Pierzchała
|
fea7a5b73f | feat: expose daemon embedding and Android ADB APIs (#480) | ||
|
Michał Pierzchała
|
72ba612e83 | fix: centralize Android adb execution (#478) | ||
|
Michał Pierzchała
|
d77a9211ab |
feat: add Android snapshot helper (#454)
* feat: add android snapshot helper * fix: harden android snapshot helper packaging * fix: harden android xml attribute parsing * fix: capture android helper window roots * fix: validate android helper install args * fix: harden android helper manifest and traversal * refactor: drop unused helper timeout parameter * feat: bundle android snapshot helper in npm package * feat: enable bundled android snapshot helper by default * refactor: simplify android snapshot helper resolution * fix: harden android snapshot helper artifacts * refactor: split android snapshot helper modules * fix: use type-only snapshot helper imports |
||
|
Michał Pierzchała
|
7c5b7670c8 | feat: add skillgym tests (#453) | ||
|
Michał Pierzchała
|
f07e82e308 |
chore: add fallow quality gate (#449)
* chore: add fallow quality gate * fix: refresh pnpm lockfile |