2 Commits

Author	SHA1	Message	Date
Michał Pierzchała	ac9e4d0f04	test: measure oracle liveness suite-wide; pin the one dead-path oracle (#1679) ... * test: measure oracle liveness suite-wide; pin the one dead-path oracle - docs/agents/oracle-negation-spike.md: assertion-negation sweep over 677 test files (5,687 verdicts). Zero vacuous tests: all 150 negation survivors decompose into assert.rejects-validator artifacts (112), helper-oracles (31), in-file-fake breakage (6), and one conditional oracle. Records the companion mock-coupled coverage-uniqueness numbers and the follow-ups they motivate (diff-scoped mutation gate, provider seam closures, transcript provenance). - watchos-sentinel: the non-watchOS test's only assertion sat in a catch block that never fires (tvOS interactor creation succeeds), so no assertion executed on the observed path. Pin creation success instead. Red-run proof: the old shape survived the negation sweep; the new shape fails under it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test(android): inject fake adb through the provider scope, not PATH stubs Adds withFakeAdb to test-utils: a scripted in-process AndroidAdbProvider installed through the production withAndroidAdbProvider seam — the same scope the daemon installs per request — replacing PATH-stub shell scripts that spawn a real subprocess per adb call. No PATH mutation, no spawns, no real subprocess waits. Converts settings.test.ts (15 tests, 23ms; waiver said "waits real settings-apply poll time") and notifications.test.ts (2 tests, 9ms). Assertions move from args-log regex greps to structural checks on the recorded call list; the fake receives device-scoped args with the -s serial pair stripped, so serial routing is enforced by the scoped provider matching device.id instead of asserted per call. Remaining PATH-stub files convert next; their contention-retry waiver entries lift together with the conversions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test(android): convert device-input-state to fake adb provider injection 10 PATH-stub cases move to withFakeAdb through the production provider scope; the 2 tests that already inject an executor directly are unchanged. Cross-invocation shell STATE_FILE state becomes a closure boolean; args-log regex asserts become structural checks on recorded calls. 12/12 green at 386ms — the residue is dismissAndroidKeyboard's two fixed 120ms retry sleeps, not stub subprocess waits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test(android): convert app-lifecycle-install adb stubbing to fake provider The adb half of every case moves to withFakeAdb through the production provider scope; installs take the documented exec-shaped fallback (exec(['install','-r',...])), matching what the PATH stub saw minus the serial pair. bundletool/zip/unzip stay real or PATH-stubbed — they run via runCmd outside the adb seam, so this file remains in the serialized subprocess-stub lane with its waiver reason to be corrected from adb to bundletool. 13/13 green at ~130ms; no case enters a retry/poll loop. Conversion note: manifest identity's `unzip -p` failure is silently swallowed (readZipEntry catch -> undefined, aapt fallback) — an invisible degradation path worth a future explicit diagnostic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test(android): convert input-actions adb stubbing to fake provider 9 PATH-stub cases move to withFakeAdb; the 3 tests already injecting providers directly are unchanged. Chunked shell-input assertions become ordered deepEqual on the recorded calls; never-called negatives and call-count checks preserved 1:1. 12/12 green. File time drops to 2.2s, all of it production sleeps: verifyAndroidFilledText unconditionally waits its [0,150,350]ms verification cadence even when the first inspection matches, so each fill verification pass costs ~500ms with an instant fake. A budget-derived cadence there (testing.md pattern 1) would put this file near 25ms; flagged as follow-up rather than changed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test(android): extract shared oracles; make fake adb failure-faithful Three test-utils extractions applied across the six converted files: - assertRejectsAppError collapses the hand-rolled AppError code+message rejection validator (10 sites here; ~30 more repo-wide can adopt it incrementally). Validators asserting details or multiple differently- flagged regexes stay explicit on purpose. - withFakeAdb gains a `provider` option for extra capabilities (snapshotHelperArtifact, reverse, ...), replacing input-actions' nested re-scoping bridge. - withFakeAdb now mirrors the local executor's contract: a scripted nonzero exit throws androidAdbResultError unless the call site passed allowFailure. Provider-scoped exec bypasses exec.ts's throw-on-close- failure, so returning {exitCode:1} took a different production path than the PATH-stub `exit 1` these fakes replaced. All 75 tests hold under the corrected semantics. Also swaps settings' inline emulator DeviceInfo literals for the shared ANDROID_EMULATOR fixture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test: lift five converted Android files from the contention-retry waiver settings, notifications, device-input-state, input-actions, and app-lifecycle-open no longer stub binaries on PATH or spawn subprocesses, so their contention mechanism is gone: they leave CONTENTION_RETRY_FILES and, through the derived SUBPROCESS_STUB_TESTS constant, the serialized subprocess-stub project (17 -> 12 files). app-lifecycle-install stays with its reason corrected: adb is now in-process, but bundletool stays PATH-stubbed and zip/unzip spawn for .aab packaging paths. Full unit suite green at the new membership: 638 files, 5,724 tests, with the five files running at unit-core's default parallelism. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test: apply review findings to the fake-adb conversion batch - app-lifecycle-open: the missing-package launch failure returns {stderr, exitCode: 1} and lets withFakeAdb's throw path produce the production-shaped androidAdbResultError instead of hand-modeling the thrown AppError — the drift the helper exists to eliminate. - withScriptedAdb deleted: the six converted files were its only callers, and a live PATH-stub export invites new tests back into the serialized lane this batch shrank. withMockedAdb stays (dispatch and runtime-hints tests still stub other binaries). - android-snapshot-helper gains androidSnapshotHelperScriptResponse so the version-probe detection and versionCode reply have one source of truth; input-actions' local copy delegates to it. - withFakeAdb's provider option becomes a distributed Omit over the AndroidAdbProvider union, so touch without gestureViewport is a compile error at the fake's boundary (planted and verified) instead of a TypeError inside production gesture planning. - spike-doc re-run checklist restores wider than the codemod globs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test: drop the consumer-less FakeAdbScript barrel re-export Fallow's dead-code gate flagged it: scripts are always passed as inline lambdas, so only FakeAdbResponse needs a name at the barrel. The type stays exported from fake-adb.ts where the withFakeAdb signature uses it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test(apple): inject fake xcrun through the tool-provider scope, not PATH stubs withFakeAppleTool mirrors withFakeAdb for the Apple seam: a scripted provider installed via the production withAppleToolProvider scope, flat simctl/devicectl invocations recorded exactly as the PATH-stub shell scripts saw them, throw-on-nonzero fidelity matching exec.ts unless the call site passed allowFailure, and the canned `simctl privacy help` listing served by default (the block withMockedXcrun injected into every script). screenshot-status-bar.test.ts converts as the exemplar: 3/3 green at 9ms with deepEqual call-sequence assertions replacing the args-log regexes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test(apple): convert apps.test.ts xcrun stubbing to fake tool provider All withMockedXcrun scripts and hand-rolled PATH stubs move to withFakeAppleTool; args-log regexes become structural call assertions (exact deepEqual where order is deterministic, presence checks where the 5s simulatorBootedMemo TTL makes boot-probe order test-dependent). 12 hand-rolled AppError validators collapse into assertRejectsAppError. 54/54 green; file test time 1172ms -> ~400ms with no test over 201ms. Five .ipa install tests keep a minimal PATH stub for unzip only: install-artifact.ts:112 and install-source.ts:438 call runCmd('unzip') directly, outside the Apple tool provider seam — the file therefore stays in the serialized subprocess-stub lane with its waiver reason corrected from xcrun to unzip. Also observed: getSimctlPrivacyServices caches per PATH+simulatorSetPath and simulatorBootedMemo keys on deviceId|setPath, so neither cache accounts for the provider scope — worked around per test, follow-up worthy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf * test: lift six Apple waivers; fix format and fallow findings from CI - interactions, simulator, screenshot, physical-device-screenshot, devicectl, and screenshot-status-bar leave CONTENTION_RETRY_FILES: the first five stopped stubbing PATH binaries in earlier refactors (measured 3-64ms per file, no subprocess activity), and screenshot-status-bar now injects through the fake tool provider. apps.test.ts stays with its reason corrected to the unzip PATH stub (xcrun is in-process; install-artifact.ts:112 / install-source.ts:438 call runCmd('unzip') outside the Apple seam). Serialized lane 12 -> 6. - oxfmt: fake-apple-tool.ts and contention-retry.ts were pushed unformatted (local check piped through tail masked the failure). - fallow complexity: the three fake-script arrows in apps.test.ts drop under threshold via shared predicates (isSimctlMainScreenScale, isSimctlScreenshot, isDevicectlDevice), which also deduplicate the screenshot pair. Full unit suite green at the new membership: 638 files, 5,724 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-08 09:01:05 +02:00
devin-ai-integration[bot]	885c1486bb	test(ci): single-retry policy for enumerated contention-flaky files (timeouts only) (#1448) ... * test(ci): single-retry policy for enumerated contention-flaky files * fix: satisfy fallow * test(ci): read failures through a lane reporter so timeouts stay distinguishable * test(ci): cover the lane reporter and drop its duplicated boilerplate * chore(fallow): own the retry lane's tool-loaded export seams * test(ci): block retries on non-test failures and classify timeouts structurally * test(ci): decide retry eligibility from runner metadata and route gate verdicts through blockers * refactor(ci): name the retry policy's rules in code instead of comments * test(ci): mark runner-aborted timeouts inside the runner instead of inferring them * test(ci): make timeout provenance a per-run secret, not a writable flag Cover direct task.meta mutation in the real child-Vitest fixture gate. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ci): retry the failed files in the first run's project and coverage modes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: drop deleted repo-health file from the retry list after #1480 Rebase onto main post-#1480: the SkillGym/repo-health descope deleted scripts/repo-health/run.test.ts, whose CONTENTION_RETRY_FILES entry would now fail this PR's own missing-file check, and inlined the slow-test budgets into the reporter, resolving the budgets-module import. Envelope comments now point at scripts/lib/lane-envelope.ts instead of the closed #1430. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-07-29 12:53:33 +02:00