* test: nightly parser fuzz lane with typed-AppError invariant (#1414)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fuzz): run envelope, artifact promotion, and harness self-check tests (#1414)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fuzz): shared scheduled-lane envelope on every terminal path, watchdog after ready (#1414)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fuzz): envelope for malformed options; add scheduled-lane health consumer (#1414, #1430)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lanes): actions:read scope, terminal error envelope, first-due grace (#1414, #1430)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lanes): anchor first-run grace to schedule registration, use exec helper in tests (#1414, #1430)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lanes): portable POSIX pickaxe pattern for schedule registration (#1414, #1430)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(fuzz): fast-check generators over the shared hazard list, drop the bundled lane-health work (#1414)
- Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own
deliverable and collides with PR #1439's implementation of the same lane. What this lane owes
(a per-run envelope) moves into scripts/fuzz/envelope.ts.
- Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries
sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a
failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string.
- Route harness.test.ts into the serialized subprocess-stub project.
- Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a
case budget far below real worker startup.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fuzz): replay the regression corpus through the worker watchdog (#1414)
A promoted hang case used to wedge the unit job until the CI timeout, because corpus replay called
checkCase in-process. It now goes through the same worker-backed watchdog the nightly lane uses, so
such a case fails against a 5s per-case budget; the file moves to the serialized subprocess-stub
project with the rest of the worker-driven fuzz tests.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fuzz): let the watchdog outlive vitest's default case timeout (#1414)
A wedged parser was surfacing as a bare 'Test timed out in 5000ms' instead of the named hang:
failure that says which input wedged, because the file's vitest timeout was shorter than the
watchdog budget times the number of replayed cases.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fuzz): complete drift provenance in the lane envelope (#1414)
configHash now covers every input that decides what a seed generates (generate.ts and the shared property arbitraries, not just the arbitraries/targets/invariant), and tool records fast-check's installed version. A generation-loop edit or a fast-check upgrade previously changed the case set while the envelope looked unchanged. A test recomputes the hash with each input omitted so a future omission fails.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): seeded concurrency torture lane for session/lease/lock invariants
Refs #1416
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): drive torture lane through real lock plan + review fixes
- derive each op's lock plan from production resolveRequestExecutionLockKeys
via a fake device-inventory provider, so reverting the router's same-device
serialization trips the overlap invariant (verified)
- assert exact replay: full scheduler trace, terminal outcome, contention
- assert real same-device lock contention in the sweep + a forced 2-client case
- split harness into bindings/invariants/envelope modules (all <500 LOC)
- emit #1430 scheduled-lane envelope (schema/SHA/hash/seed range/duration/result)
and upload it from the nightly workflow
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): pass claim data as plain view accessors (fallow)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): gate lock plan on shouldLockSessionExecution; sweep replay + forced-device contention; whole-lane envelope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): add real-scope runLocked serialization guard; whole-lane envelope duration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): allow seed 0 replay; add seed-0 regression (TORTURE_SEED must accept 0)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* obs(#1430): add scheduled-lane freshness/cadence health watcher
Discovers schedule: workflows from .github/workflows/, reads recent
scheduled runs via the GitHub API, and opens/pings a tracking issue when
a lane misses or fails two consecutive cadences. Pure model unit-tested
and gated on PRs; API I/O + issue open/ping run nightly.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): give scheduled-lane watcher a two-cadence newborn grace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): anchor lane grace on schedule-introduction, not workflow age
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): derive schedule-activation semantically via git, through runCmdSync
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): add merge-commit regression pinning first-parent + committer time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): pin scheduled-lane-health issue-write route via stubbed fetch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): unbundle #1430 watcher; make torture lane nightly-only
Strip the scheduled-lane-health watcher (scripts + workflow + PR gate) —
it is #1430's deliverable and collides with PR #1438's workflow of the
same filename; keep only this lane's #1430 envelope writer.
Move the torture lane under test/integration/nightly/ so it is out of the
test:integration:node glob, and run it via an explicit, disclosed PR step
plus the nightly sweep.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: fix stale torture-lane paths after nightly/ move
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(daemon): adopt shared lane-envelope for torture lane
Rebase onto main (post-#1441) and replace the lane-local LaneEnvelope
dialect with the shared scripts/lib/lane-envelope.ts builder, so the
#1430 health watcher parses one schema: commitSha->commit, sourceHash->
configHash, seedRange/runs moved into the typed data payload, and the
sweep encoded as seed "<start>-<end>".
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: ratchet mutation score over enumerated decision kernels
Adds a Stryker (vitest runner) mutation lane scoped to the decision kernels,
a per-module baseline with tool/config provenance, and a ratchet that only
lets scores rise. Non-gating until two consecutive stable weekly sweeps.
Refs #1415
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: declare the mutation test-scope seam for production-export analysis
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: own kernel tests in the mutation registry and ship the #1430 lane envelope
- restore bench:help-conformance, broken by a formatting-path edit
- kernel test files select their module on PRs (registry `tests` + workflow paths),
asserted to reach the kernel through the import graph
- every mutation run writes the standard scheduled-lane artifact envelope
- move src/utils/__tests__/errors.test.ts beside its source per the mirror rule
Refs #1415, #1430
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: derive kernel test ownership and land the scheduled-lane health monitor
Ownership of a kernel's tests is now computed from the static import graph
(scripts/mutation/ownership.ts) instead of a hand-listed set, so a test that
reaches a kernel indirectly -- src/__tests__/daemon-error.test.ts through
src/daemon.ts -- selects that kernel on a PR. The PR lane triggers on every src
test and shards the derived modules, keeping wall clock at one module.
The lane envelope (#1430) is now written on every exit path with the stage it
reached, so a crash before any mutant runs is distinguishable from a lane that
never ran. Adds the derived cadence monitor (scripts/lane-health, daily
workflow): scheduled lanes are enumerated from .github/workflows/ and reported
dark, failing, or never-run against their own cron cadence.
* fix: merge only Stryker reports from a shard directory
The shard artifacts now carry the lane envelope beside mutation.json, and the
merge globbed every .json under the download path, so the ratchet job fed the
envelope to the report parser and died after the mutants had already run.
* fix(mutation): fail on incomplete shard sets and envelope pre-run failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mutation): downgrade a passing envelope when a later lane step fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mutation): shard by registry, defer the PR lane, drop the bundled watcher
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: describe registry sharding and the deferred PR mutation lane
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mutation): make the pre-graduation tooling exception select real mutants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mutation): give the worktree fixture commits their own identity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: frozen replay-compat corpus with expected verdicts (#1417)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: pin replay-compat corpus bytes to released blobs and assert via parseReplayInput
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: lock replay-compat provenance kind by corpus area and verify it in CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: describe corpus provenance-kind lock and CI job
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: prune replay-compat corpus to minimal witnesses per shipped form
Reviewer feedback on #1436: the mechanism earns its place, the dataset did not.
Drop the 30 corpus entries whose bytes repeat a syntactic form or a migration
refusal another entry already witnesses (platform twins and adjacent-release
re-recordings), leaving 22 deliberate entries; make note required and state per
entry which form or refusal it is the sole witness of.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: address corpus review nits (typed coverage list, cap rationale, derived-citation note)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: split corpus rule — form from the release, verdict from today's parser
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: format corpus README emphasis markers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: property-based testing foundation over parse/print and geometry kernels (#1413)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: derive selector and replay command domains from their owning registries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bench): renderer-pinned samples, topic-coverage gate, error quizzes; trim skillgym to agentic checks
The help conformance bench's quoted CLI output is now sourced from
scripts/help-conformance-sample-outputs.mjs, and every sample is rebuilt
through the real production renderers (settle output formatters,
printHumanError, formatSnapshotText, refMutationAdmissionResponse) by
scripts/__tests__/help-conformance-sample-outputs.test.ts — a rendering or
message change fails deterministically instead of leaving the bench grading
against output the CLI no longer prints. This retires the fabricated
recoverable-failure envelope (production never throws a textual settle
timeout; that case is replaced by a real DEVICE_IN_USE recovery quiz).
Bench cases move to scripts/help-conformance-cases.mjs and are enumerated
against the help-topic registry: helpTopicIds() is exported from cli-help,
and scripts/__tests__/help-conformance-topic-coverage.test.ts fails when a
help topic has neither a bench case nor an explicit waiver. New case
families: error-envelope recovery quizzes (device-in-use, stale pinned ref,
ambiguous find match, app-not-installed) pinned to real error text, topic
coverage for tv/web/react-native/debugging/workflow, and a metamorphic twin
of the settled-diff quiz.
The skillgym smoke suite shrinks from 119 cases to the 5 that measure what
only an agentic runner can show: skill routing plus output interpretation
with a proven local CLI help probe (local-cli-help-policy). Its embedded
samples now import the same pinned constants, replacing hand-transcribed
output that had already drifted from the renderer. Knowledge checks belong
to the bench; live fixture behavior belongs to the iOS simulator e2e suite.
* review: drive error samples through the real producers; enforce local-help on the routing smoke
The DEVICE_IN_USE, AMBIGUOUS_MATCH, and APP_NOT_INSTALLED parity tests no
longer hand-author the producer message before rendering: each drives the
actual producer — buildDeviceInUseBySessionError (extracted in
session-open.ts and called by the handler), buildAmbiguousMatchError (now
exported from find.ts), and buildAppNotInstalledError (extracted in
app-resolution.ts and thrown by the resolver). Because each factory is
exported from its producer file and called by the production path, dropping
the production call would make it test-only and fail
check:production-exports — the wiring is gate-enforced, not conventional.
open-and-snapshot now sets requireLocalCliHelp and
allowOnlyLocalCliHelpCommands, so the 'skill plus local help' claim is
observed rather than assumed; without them the case can pass on model prior
alone.
* docs: restructure AGENTS.md and CONTEXT.md for progressive disclosure
Apply the Claude 5 context-engineering guidance to the repo's agent docs:
keep the always-loaded file to gotchas and invariants, and move situational
guidance one hop away behind a routing table.
AGENTS.md 315 -> 229 lines. Cut generic agent-behavior boilerplate, three-way
duplication (Common Mistakes restated Hard Rules; Finding Source Owners
restated the registry section), and facts visible from the repo itself.
Kept verbatim: the expensive-lessons principles, enforcement gates, Hard
Rules, and environment traps.
Split out docs/agents/{cli-flags,pull-requests,device-verification}.md and
folded the Testing Matrix into docs/agents/testing.md, reframed around
pnpm check:affected so the prose stops duplicating the selector.
CONTEXT.md keeps all 50 terms, now grouped under a section index so a task
loads one section instead of the whole glossary.
* fix(check-affected): move the selector-owning sentinel to the Testing Matrix
The Testing Matrix moved from AGENTS.md to docs/agents/testing.md, but the
affected-check selector still treated only AGENTS.md as selector-owning. A
later matrix edit would have been classified as inert docs and skipped the
fail-open, so the selector could keep deriving gates from a spec that had
changed underneath it.
Move the sentinel with the prose, as a named SELECTOR_OWNING_DOCS set so the
next move is one line, and fix the two in-code comments plus the testing.md
paragraph that still pointed at the AGENTS.md matrix.
* docs: restore two rules dropped by the AGENTS.md split
Review caught two repo-specific rules that did not survive the move. Both are
prose without any backticked identifier, so the identifier-diff used to verify
the split could not see them.
- "Test through public interfaces; do not add unrelated production exports
solely to enable tests" returns next to the behavioral-tests rule in
docs/agents/testing.md, with the reason it exists.
- The guidance-ownership rule (decide whether new guidance/schema/metadata
belongs to the command surface, CLI grammar, CLI help, MCP projection, or
daemon runtime) returns to the always-loaded Docs & skills section, since it
governs all command-surface work and not just the flag case.
Also point the ADR routing row at docs/adr/README.md, which is already the
"read when you touch…" index, rather than at the bare directory.
* chore: close out architecture experiments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: record unavailable live experiment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: make Android perf script atomic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: explain atomic perf workflow
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: simplify back-edge diagnostics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: record contract and digest spike selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: project navigation client and MCP contracts from executable definitions
Collapse the three independent per-command projection declarations (facet
clientMethod, public client method signature, MCP output schema) for the typed
system navigation subset (home, back, rotate, app-switcher, tv-remote) onto a
single colocated projection in src/contracts/navigation.ts. The family builder,
public client type, and MCP schema map now derive from those five projections.
Refs #1185
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: add opt-in network response digest
Preserve every network entry and top-level recovery/actionability signal while
dropping only verbose per-entry header, body, and raw-log fields at digest
response level. Record deterministic output-economy baselines and parity tests.
Refs #1186
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: add derived fail-open check:affected selector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: simplify selector for complexity gate; add docs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: fail open on ambiguous non-source fixtures; guard catalog against real package.json/vitest.config
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: use src/utils/exec.ts process helpers in check:affected runner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(check:affected): SkillGym ownership, honest catalog, working-tree discovery
- Add SkillGym ownership for skills/ and test/skillgym/; stop short-circuiting
their Markdown as docs-only (findings 2 & 4).
- Drop the fabricated GitHub 'SkillGym' job: it is a local-only gate, now
localRunnable with no CI job, guarded by a workflow-existence self-test (3).
- Fold working-tree (staged/unstaged/untracked) state into local discovery and
disable rename detection so both rename paths classify (1).
- Add run.test.ts entrypoint regressions (real diff/status/rename discovery,
--run order/skip/stop-on-failure).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(check:affected): union staged + unstaged diffs so they cannot cancel
A single `git diff HEAD` nets index against working tree, so a staged add
and an unstaged delete of the same file cancel and hide it. Collect
`--cached` (staged) and unstaged diffs separately and union them; add a
cancellation regression test.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(check:affected): cover required suite gates
* refactor(check:affected): delegate tests to vitest
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: slow-test ratchet, budget-derived emulator poll, speed guidance from experiments
Measured (2026-07-04, full unit suite: 340 files / 3,210 tests / 48s wall):
wall clock was bounded by the slowest FILE (44.6s android monolith at ~7x
file-level parallelism), and the slowest tests were sleeping through real
production budgets (10.8s proving 'times out' by waiting the constant out,
8s emulator polls at 1Hz, real retry backoff). Two config experiments
rejected with data: --no-isolate exploded the suite to 205s (module state
thrashes across files sharing workers) and --pool=threads changed nothing.
- scripts/vitest-slow-test-reporter.ts: the slow-test ratchet. Unit budget
2.5s / integration 15s; failure at 2x budget (the band between reports
without failing so host-load variance cannot make the gate cry wolf);
36 pinned offenders, exact keys, ratchet-only pin (tracking #1098).
- waitForAndroidEmulatorByAvdName: poll cadence derives from the caller's
budget (min 1s, floor 50ms, ~timeout/20) — devices.test.ts 25.6s -> 2.8s
(9x) in isolation, and short-budget production calls stop sampling at
1Hz against small budgets.
- vitest.config: slowTestThreshold 500 for local visibility; reporter
wired; isolation/pool decisions documented with the measurements.
- docs/agents/testing.md 'Speed rules' + AGENTS.md testing bullet: the
three conversion patterns in preference order (budget-derived cadence,
budget-wiring assertion, fake clocks), the no-seam constraint, and the
file-granularity Amdahl argument that makes the monolith test split a
wall-clock fix, not just navigation.
* fix: fallow findings on the slow-test gate — import edge, factory reporter, unit tests
The string-path reporter wiring read as a dead file (fallow cannot see
vitest's reporter loading); the config now imports the factory, making
the edge real and type-checked. The class shape tripped the
unused-class-members rule (framework callbacks are invisible to
reference analysis) — converted to a factory returning the Reporter
object, with the classification and rendering logic extracted as pure
exported functions. Those functions now carry their own unit tests
(budget bands, integration budgets, pin matching, warn-vs-fail
rendering), which also grounds the CRAP estimate in real references.
Canary re-verified: unpinned 5.2s sleeper fails the run with exit 1;
clean runs exit 0.
* feat: expose web network dump through agent-browser
* fix: reduce web network mapper complexity
* fix: gate web network headers by include mode
* test: assert compact web network summary
* refactor: simplify web network dump mapping
* refactor: trim web network coverage
devin-ai-integration[bot]