* perf(package): strip comments from the Apple runner source the npm package ships
The packager copies apple/runner/** into dist/ as Swift source, removing only
its AGENT_DEVICE_RUNNER_UNIT_TESTS blocks, so doc comments and design notes were
downloaded on every install: 71.9 kB of 441.2 kB of packaged runner Swift.
Add a lexical scanner for the removal. A regex cannot do this: `//` and `/*`
open a comment only in code position, raw literals move their own delimiter and
escape with the `#` count, interpolation segments hold code and further
literals, and Swift block comments nest. A construct the scanner cannot account
for throws at packaging time instead of shipping Swift that does not compile.
* fix(package): keep Swift regex literals out of the comment scanner
`#/foo//bar/#` is a valid extended regex literal with no comment in it, but the
scanner only knew the `#"` raw-string family, so it read the literal's `//` as a
line comment and shipped `let pattern = #/foo` — Swift that does not compile.
Add `#/…/#` and `##/…/##` as a literal context: matching `#` counts, the
single- and multi-line forms, Swift's own-line rule for a multi-line closing
delimiter, and the `\/` escape that keeps one from closing early.
Bare `/…/` literals stay unresolvable, because the same `/` opens a comment,
divides, and starts a regex literal, and only the parse separates them. Where
one could begin — an expression position whose `/` is not followed by a space,
a tab or `)` — packaging throws by file and line instead of rewriting bytes it
cannot prove are code. Divisions (`width/2`, `Double(3)/Double(4)`), the
recording scripts' shebang and `(/)` keep flowing through.
* fix(package): keep the packaged runner source on the checkout's line numbers
`dist/apple/runner/**` is the Swift a user's `xcodebuild` and the runner name a
file and line in (it lands in runner.log), so those numbers are only worth
reading if they point at the same line of `apple/runner/**`. Both rewriting
passes now empty the lines they remove instead of deleting them: comment removal
(889 lines, 889 B) and the pre-existing unit-test `#if` block strip, which was
moving everything below a block by up to 883 lines (3,737 lines, 3,737 B).
`dist/apple/runner/` 555,907 B -> 488,635 B (-67,272 B, -12.1%); its Swift alone
441,196 B -> 373,924 B (-15.2%). Parity costs 4,626 B of the 71,898 B the
previous head saved.
Nothing in the repo compiles the packaged source, so a mis-lex that failed to
throw would ship Swift that does not build and no gate would see it. Add
`pnpm check:packaged-runner-swift`: it packages into a throwaway root and asserts
line-count parity plus the line of every declaration each packaged file still
carries, then runs `swiftc -parse` over all 44 files. The parse half reports
itself skipped where no Swift toolchain exists, so the gate is declared on the
macOS lane, where both halves run.
* fix(ci): stop ten artifact uploads discarding their hidden paths
`actions/upload-artifact` has excluded hidden files and directories by default
since v4.4 (this repository pins v4.6.2), and most diagnostics here are written
under `.tmp`. Ten upload steps across seven files therefore uploaded nothing
from those paths: macos.yml's xcresult bundle, both mutation lanes' reports and
shards, replays-nightly's fuzz output, xctest-nightly's results,
test-app-build-cache's fixture tarball, and 1874-diagnose's per-iteration logs.
Most fail silently, since they pair the omission with `if-no-files-found: warn`
or `ignore`. test-app-build-cache sets `error`, so that one does not.
A structural guard rather than a shared upload wrapper: the wrapper would be a
shallow mirror of the action's options over artifacts with different owners,
while the policy question — a hidden path needs the flag — is one rule that
belongs in one place. Each workflow still declares its own artifact.
test/ci/upload-artifact-hidden-paths.test.ts holds it across every workflow and
composite action, and is red if any single flag is dropped.
* test(ci): scan every YAML shape GitHub accepts, not just top-level *.yml
The guard read `.github/workflows/*.yml` and assumed local actions live one
directory deep as `action.yml`. GitHub also reads `.yaml` for both, and local
actions nest, so a hidden-path upload in any of those shapes passed the gate.
It now walks the `.github` tree recursively for either extension, and a second
test plants the three shapes the old scan missed and asserts all three are
found — executable rather than a one-off manual check. Red against narrowing the
extension, against dropping the recursive walk, and against removing any single
real flag.
Also drops the action-version note under the comment rule in #2087; the version
behaviour belongs in the PR, and the assertion message already says what the
omission costs.
* ci: consolidate CI workflow from 15 jobs to 8
Merge single-gate ubuntu jobs into grouped jobs sharing one checkout and
install: Lint & Format (plus the static text assertions), Repo Guards
(layering/selector/wiring/maestro/mcp-metadata), Compatibility &
Provenance (shared fetch-depth: 0 checkout), Typecheck & Package, and
Integration Tests (absorbs the web smoke with step-scoped env). Every
gate remains an independently named run-gate step; the gate manifest
derives lane ownership structurally.
Drop the Bun setup from FreeRange: @chenglou/freerange's bin is a plain
Node script. It stays GitHub-owned; only the runtime requirement is
retired.
* ci: fold FreeRange into Repo Guards and skip no-op fixture release jobs
FreeRange runs on plain Node now, so its gate joins Repo Guards as the
last step instead of occupying its own worker for the slowest guard.
The fixture release matrix filters to entries that will actually build,
so a cached-fingerprint PR starts zero release runners.
* ci: fold host XCTests into the macOS smoke lane and shard Coverage
The macOS lane now builds one unit-test-flagged runner bundle that both
the host XCTest run and the replay smoke consume, so the host lane no
longer occupies its own macos-26 runner behind a separate queue. The
host lane's file moves with it, and check:xctest-selection follows.
Coverage shards across two runners via blob reports and merges them on
a report job that evaluates thresholds once over the full suite and
produces every coverage artifact. The tmpdir leak check runs per shard,
since a leak lands on whichever runner executed the file.
* ci: drop local shard-smoke artifacts from tracking
* ci: enforce coverage thresholds only on the merged run
A shard evaluates its own half-suite coverage, so the global gate fired
per shard. Shards now report without gating; Coverage Report keeps the
real thresholds over the full merged suite.
* ci: include hidden files when uploading coverage blobs
* ci: skip device lanes for root-level docs-only changes (#1781 A9)
Add AGENTS.md, CHANGELOG.md, CONTEXT.md, CONTRIBUTING.md, LICENSE, and
SECURITY.md to the pull_request paths-ignore block in ios.yml,
android.yml, linux.yml, macos.yml, ci.yml, and size.yml. These
root-level docs files were the only gap left after docs/**,
website/**, and README.md — PRs #1568 (SECURITY.md only), #1697
(CONTEXT.md + docs/adr only), and #1722 (AGENTS.md + docs/) each still
triggered a full 9-15 min macOS iOS run despite touching only prose.
Why each file is safe to ignore for every one of these six workflows:
- None of the four device workflows (ios/android/linux/macos) or their
composite actions read any of these six files at runtime; the only
hits from `grep -rln` across scripts/, src/, test/, and
.github/actions/ are prose comments pointing humans at CONTEXT.md or
AGENTS.md sections (e.g. scripts/layering/check.ts,
scripts/wire-compat/run.ts, src/mcp/tool-ref-pins.ts) — never an
`fs.readFileSync`/`readFile` of the file itself.
- The check-affected selector (scripts/check-affected/model.ts)
already classifies all six as pure docs: `isDocs()` matches any
`.md` file plus the literal `LICENSE`, and `docsOwnership()` only
special-cases `website/docs/docs/commands.md` (unrelated). So these
files already select zero checks — they only ever produced
`docsOnlyPaths` entries, never `SelectionReason`s.
- Because they select zero checks, the gate-manifest's path-coverage
category derivation (`scripts/gate/model.ts` `categories()`, which
iterates `plan.reasons`) never records a category for them, so
ci.yml has nothing check-manifest-only that these six files would
need to keep reachable. `pnpm check:gate-manifest` and
`pnpm check:gate-manifest:test` both stay green after the change
(48 checks / 33 lanes, 28/28 gate tests passing).
- size.yml's bundle-size job (scripts/size-report.mjs) measures the
`pnpm build` dist output and startup timing only — no reference to
any of these six files. (npm packs LICENSE/README.md into the
publishable tarball, but that's a `pnpm check:package` node-22.12
concern in ci.yml's packaged-cli job, which is driven by `dist`
contents and `package.json`, not by LICENSE/README prose — already
evidenced by README.md being ignored here since before this change.)
Scope disclosure: `mutation-affected.yml` uses a `paths:` allowlist
(not paths-ignore) so it's structurally unaffected; `test-app-build-cache.yml`
has no path filter at all. Neither was touched.
actionlint and `pnpm check:gate-manifest`/`:test` pass on the changed
workflows.
* test: pin root-doc paths-ignore entries with a regression test
Addresses review feedback on #1791 from thymikee: the docs-only
classifier for AGENTS.md/CHANGELOG.md/CONTEXT.md/CONTRIBUTING.md/
LICENSE/SECURITY.md across ios.yml/android.yml/linux.yml/macos.yml/
ci.yml/size.yml had no regression pin. Neither check:gate-manifest
(only proves a *registered check* is reachable) nor actionlint (only
validates YAML shape) nor generic Markdown coverage would catch a
single dropped entry — e.g. LICENSE reappearing in one workflow's
paths-ignore list but not another's would silently put a full 9-15 min
device run back on prose-only PRs.
test/ci/root-docs-paths-ignore.test.ts parses the six real workflow
files and asserts, using the same matchesGlob the gate-manifest model
uses to decide lane triggering, that each of the six root docs is
ignored by each workflow's pull_request paths-ignore. Registered in
vitest.config.ts's unit-core project next to its sibling
upload-agent-device-artifacts.test.ts (parse-only, no device/subprocess
lane needed).
Verified red on main (all 36 file x doc assertions fail — confirmed via
a throwaway script reading `git show main:.github/workflows/*.yml`)
and green on this branch (6/6). Full unit-core project (873 files /
6641 tests) still passes; check:gate-manifest and
check:gate-manifest:test unchanged (48 checks / 33 lanes, 28/28).
* test(ci): prove every registered gate is owned and reachable (#1429)
A check that silently stops running looks exactly like a green build. Two
suites had already stopped: `check:tmpdir-leaks` (with its model tests) and
`test:fixture-cache` are real package scripts that no workflow ran, reachable
only through the `check:unit` aggregate CI never invokes.
`CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the
only way CI runs one, so finding what a lane runs is a scan for `pnpm gate`
rather than an attempt to interpret shell. `pnpm check:gate-manifest` then
asserts against the real workflows that every registered check is run by some
qualifying lane (per unit, not per script name), that every check the real
selector activates for a path is run by a lane that path would start (#1420's
class), and that every Vitest project and suite script belongs to a check.
The wiring that keeps those honest is asserted too: a gate id must name a
registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false`
unowns what it guards, an action declared to run a gate is proven to, and a
job whose steps the loader cannot open fails closed.
It deliberately does not try to prove CI runs project code only through
`pnpm gate`. Whether a shell block executes project code is not decidable from
its text, so shell this model does not recognise earns no ownership credit —
the failure direction is a check reported unowned, never one waved through.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
* test(ci): update the two suites that assert on rewired workflow text
`scripts/mutation/workflow.test.ts` and `test/ci/trusted-fixture-artifact.test.mjs`
read the workflow and action files and assert on their command text, so routing
those steps through `pnpm gate <id>` moved what they were matching.
They are the two suites the manifest cannot help with: it proves a gate is still
run, not that a test asserting on how CI spells a command was updated with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
* fix(ci): credit gates by execution shape, and keep every guard
Three ways the manifest could report a gate as owned when it does not run.
1. Crediting was a substring scan over `run:`, which #1429 explicitly rules
out — "do not infer reachability from a command name merely appearing in
workflow text". `false && pnpm gate x`, a gate inside `if false; then … fi`,
one named in a heredoc, and `echo pnpm gate x` all credited it. There is a
live instance: conformance-regenerate.yml's "Fail if regeneration changed
anything" step names `pnpm gate maestro-regenerate` inside an error message
telling a human to run it, and that credited the gate.
A gate now counts only as the first command segment of a line, and a body
carrying shell structure earns nothing. Reachability inside a script is not
decidable, so this does not try: unrecognised shape means no credit and the
check reports unowned. `VAR=$(pnpm gate x …)` is read, since the assignment
form is unambiguous and the gate runs.
2. Job-level `if:` was not modelled at all, though six live jobs carry one, so
a job that cannot run still credited every gate inside it. Two conditions on
the mutation lanes are now declared.
3. A caller's `if:` REPLACED the guard on a nested composite-action step
(`guard[0] ?? step.condition`), so an outer `always()` erased an inner
`if: false`. Steps carry every guard between the lane and the step.
Also corrects two source comments that still claimed project code run outside
the runner fails the manifest. It does not: such a step earns no credit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
* ci: add the run-gate action that names a gate structurally
The seam the ownership proof will read instead of shell. A lane says which
gate it runs in `with.gate`, a typed input the manifest reads straight out of
the YAML and validates against CHECK_CATALOG.
Nothing here is wired yet — the ~60 call sites and the model change follow.
Added first so the target of that conversion is reviewable on its own.
`args` cannot select which gate runs; it is appended after the id, so the
worst a wrong value does is fail the gate it already named. There is no
`|| true` and no output capture: the gate's exit code is the step's exit code,
so a gate cannot run without being able to fail its lane.
Part of #1429.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
* merge: main (#1770) and route its three new steps through the runner
#1770 landed the orphan-check fix on main, wiring `check:tmpdir-leaks`,
`check:tmpdir-leaks:test` and `test:fixture-cache` into Coverage, Layering
Guard and Integration Tests. This branch had wired the same three through
`pnpm gate`, so the merge produced two steps per check rather than a conflict
— each check ran twice.
Kept main's steps, with the placement and reasoning reviewed on #1770, and
changed only their `run:` line to the canonical runner. Dropped this branch's
duplicates. Net effect on CI is unchanged: the same three checks, in the same
three lanes, once each.
Gate manifest green after the merge: 47 checks wired across 33 lanes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
* fix(ci): address review — suite detection, freerange, glob, vacuous skip-list
Six review findings plus the mutation blocker.
[bug] `registered` was shape-only, so a `test:*` script running
`node src/bin.ts test <dir>` resolved to a `script:` leaf and was invisible.
Four `test:replay:*` scripts were owned only because someone hand-registered
them; `test:replay:android` was neither registered nor reported while the
nightly ran the same six .ad files by inlining them. A `test:*` script is now
a suite by name. `replay-android` is registered, and the nightly runs the
script instead of re-listing its files so the two cannot drift.
The nightly invokes it inside `reactivecircus/android-emulator-runner`'s
`script:` input — shell handed to a third-party action this loader does not
read — so the suite executes but cannot be credited. Recorded in
UNPROVABLE_OWNERS with that exact reason rather than assumed.
The fixed detector also found a second orphan the review did not name:
`test:integration:progress`. That one is a reporter whose `--check` sibling
is the registered gate, so it is declared in REPORTING_SCRIPTS — a
declaration that itself fails when inert.
[bug] `freerange` defaulted to localRunnable, so fail-open ran `fr` (a Bun
binary) on the pre-push path. Now false.
[suggestion] The `--run` skip-list asserted `build:android-snapshot-helper`,
a name `android-helpers` no longer uses, so it could not fail. Derived from
the catalog instead.
[suggestion] `matchesGlob` joined `**` splits with `.*`, making the adjacent
slash mandatory — GitHub's `**` matches zero directories, so
`src/**/*.test.ts` did not match `src/a.test.ts`. Pinned against
`packages/*/src/**/*.test.ts`.
[suggestion] Deleted the unwired `run-gate` action. It had no callers, was
absent from GATE_ACTIONS, and its comment described a system that had not
shipped. It returns with the rewiring, not before.
[suggestion] Collapsed the module headers that narrated discarded designs.
Mutation: `daemon entrypoint publishes HTTP metadata and cleans up on
shutdown` is the only test here that spawns a real daemon process. It takes
~1.1s alone but exceeds Vitest's 5s default inside Stryker's dry run, which
aborts the sweep before a single mutant runs. Given 30s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
* fix(mutation): order sandbox aliases longest-first so subpaths resolve
Every shard of the mutation sweep aborted in Stryker's dry run with:
Cannot find package '@agent-device/selectors/engine' imported from
.tmp/stryker/sandbox-*/src/core/selector-pipeline.ts
The alias was generated correctly; it just never won. Vite matches a STRING
alias by prefix and takes the first hit, and `workspaceSpecifierTargets`
emitted the bare `@agent-device/selectors` ahead of the subpath entries. The
bare entry therefore captured `@agent-device/selectors/engine` and rewrote it
to `…/src/index.ts/engine`, which does not exist; Node fell back to real
package resolution, could not find the subpath inside the sandbox, and the dry
run failed before a single mutant ran — so the shard uploaded an empty
envelope instead of a report and the ratchet failed for want of one.
Sorting longest specifier first makes the most specific alias win:
@agent-device/selectors/engine -> packages/selectors/src/engine.ts
@agent-device/selectors/ast -> packages/selectors/src/ast.ts
@agent-device/selectors -> packages/selectors/src/index.ts
`/ast` never tripped this because nothing in a related test set imported it;
`selector-pipeline.ts` introduced the first subpath import that mattered
(#1744), so the mutation lane has been unable to run since that landed. Any
PR touching `scripts/mutation/**` — which fails open into the full sweep —
would have hit it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
* refactor: derive gate ownership from workflow structure
* fix: run gates without optional arguments
* fix: resolve mutation workspace subpaths exactly
---------
Co-authored-by: Claude <noreply@anthropic.com>
* ci: fold single-grep jobs into steps, call named pnpm scripts
- Merge ios-runner-swift-compat and no-test-di-seams (each just
checkout + one rg assertion) into steps of a new static-checks job,
keeping each step's own failure message. Removes two job-scheduling/
checkout overheads and two PR status-check lines.
- Replace the layering-guard job's inlined copies of check:layering and
depgraph:test with the named pnpm scripts, removing the silent-drift
risk between the workflow and package.json.
- Fix the same drift in conformance-regenerate.yml, which inlined
maestro:conformance:regenerate byte-for-byte.
- Leave affected-selector's inline node invocation as-is: R8's zero-dep
closure check (scripts/layering/zero-dep-jobs.ts) finds a job's entry
scripts by matching literal paths in the run: block, so switching to
`pnpm check:affected:test` would zero out its entries and make R8
fail closed. Documented inline why this one stays inlined.
- Leave publish-mcp-registry.yml's sync-mcp-metadata --check alone: that
job never runs the setup-node-pnpm action, so pnpm isn't provisioned
there at all.
Refs #1462
* ci: teach R8 to resolve pnpm script names, drop affected-selector's inline copy
R8's zero-dep-job entry scan matched literal script paths in a run: block,
so a bare `pnpm <script>` invocation found zero entries and R8 failed
closed — the reason affected-selector kept an inline node command instead
of calling pnpm check:affected:test (#1462). zeroDepJobs now also resolves
a pnpm script name against package.json and scans the resolved command for
entry paths, so affected-selector can call the named script like every
other job.
Also replaced the other workflows' inlined copies of named package.json
scripts (test:replay:*, perf, perf:android, maestro:conformance:differential,
check:mcp-metadata, size) with their pnpm names, keeping each job's
CI-specific trailing flags — found via a repo-wide sweep for any run: block
whose text duplicates a scripts entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L67kDSTAJwaoLCwANEJFRM
* fix: revert publish-mcp-registry pnpm regression, recurse R8 alias resolution
publish-mcp-registry.yml's job only provisions Node via actions/setup-node,
never the repo's setup-node-pnpm action, so pnpm is never installed there —
the earlier sweep's `pnpm check:mcp-metadata` would have broken the release
path. Reverted to the direct node invocation with a comment explaining why,
matching the PR's own stated rationale for leaving it alone.
zeroDepJobs' pnpm-alias resolution only expanded one level: a resolved
script that itself invoked another named pnpm script had its entries
silently dropped from R8's closure. resolveRunEntries now recurses through
chained aliases with a per-chain visited set, so a nested alias's entries
are found and a cycle stops re-expanding a repeated name instead of
recursing forever. Added coverage for both the chained and cyclic cases.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L67kDSTAJwaoLCwANEJFRM
---------
Co-authored-by: Claude <noreply@anthropic.com>
* chore(deps): add Renovate config and enforce packageManager pnpm version in CI
Refs #1422
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: bump pnpm to 11.17.0 and format the whole repo with oxfmt
format/format:check drop their hand-maintained path list: oxfmt already skips
node_modules and honors .gitignore, so the only exclusion list is
.oxfmtrc.json ignorePatterns.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mutation): accept either quote style in the affected-lane path filter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): keep fixture-app runtime deps as individual Renovate PRs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: remove JS test wrappers, run .ad replay suites directly on CI
The platform JS test files (ios.test.ts, android.test.ts, macos.test.ts)
were thin wrappers that shelled out to `agent-device test` and asserted
on JSON counts. Since the CLI already exits non-zero on failure, the
wrappers added no value. CI now invokes the replay suites directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add local replay entrypoints and restore physical device test
- Add pnpm scripts for each replay suite (test:replay:ios,
test:replay:ios-device, test:replay:android, test:replay:macos)
so local runs still exercise platform replays
- Restore physical-device iOS replay step in CI workflow
(conditional on IOS_UDID variable)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Michał Pierzchała