mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
Michał Pierzchała
ac52281448
* fix(test): deterministic temp-dir cleanup across node --test lanes node --test has no global setup/teardown hook, so unlike Vitest (#1593) every node --test package.json script (maestro:conformance, mutation:test, check:affected:test, check:coverage-changed:test, check:layering, depgraph:test, check:tmpdir-leaks:test, check:contention-retry, test:fixture-cache, test:smoke(:web), test:integration:node, test:concurrency-torture) still created scratch directories against the real, unredirected os.tmpdir(), with cleanup only as reliable as each call site's own try/finally — which a crash, OOM, or timeout kill bypasses entirely. Add scripts/node-test-tmpdir.ts: it wraps the whole `node --test` invocation as a child process, redirecting TMPDIR to one disposable, pid-tagged directory (shared root/prefix with the Vitest lane) and removing it from the process 'exit' event, which fires on normal completion, a thrown error, or a forwarded SIGINT/SIGTERM alike. Every node --test script now runs through it. check-tmpdir-leaks.ts already scans by root/prefix, so it covers both mechanisms with no changes to its detection logic. Verified: a node --test process that mkdtemp's then gets SIGKILL'd leaves a directory behind unwrapped; wrapped and SIGTERM'd, TMPDIR is redirected and the directory is gone with no orphaned processes. All 13 wrapped lanes and the full Vitest suite (5,591 tests) pass with zero residual agent-device-test-run-* directories after the run. Fixes #1595 * test(tmpdir): ratchet every node --test script through the wrapper The 13 lanes wrapped in package.json were a one-time hand sweep with nothing enforcing the pattern going forward — a 14th node --test script added later without scripts/node-test-tmpdir.ts would silently reopen #1595 for that one lane. Add a structural check to scripts/node-test-tmpdir.test.ts (now part of check:tmpdir-leaks:test) that reads package.json and fails if any script invokes `node ... --test` without routing through the wrapper. Dumb string matching over the scripts map, no shell parsing, with an explicit (currently empty) NODE_TEST_WRAPPER_BYPASS_ALLOWLIST for any lane that must legitimately bypass it. Verified it both passes on the current package.json and fails when a synthetic unwrapped `node --test` script is added. * fix(test): preserve the Swift cache and close the raw node --test bypasses Review on #1661 found two gaps: 1. The wrapper only overrode TMPDIR, so it discarded and forced a recompile of the durable Swift compiler cache every run instead of mirroring vitest-tmpdir-global-setup.ts's carve-out for it. Read os.tmpdir() before the child's TMPDIR redirect takes effect and set AGENT_DEVICE_SWIFT_CACHE_DIR from that (only when unset), same as the Vitest lane — the two now share one durable cache instead of each discarding and recompiling their own. Added a probe assertion (scripts/node-test-tmpdir.test.ts) that fails without the fix and passes with it (verified both ways). 2. docs/agents/testing.md documented raw `node --test` commands for the iOS smoke files, and the android/ios/conformance-regenerate/nightly workflows invoked `node --test` directly outside package.json. Routed all of them through scripts/node-test-tmpdir.ts so the documented local commands and CI lanes get the same crash/timeout-safe cleanup the package.json scripts already have.
67 lines
2.8 KiB
YAML
67 lines
2.8 KiB
YAML
name: Conformance Regenerate Verify
|
|
|
|
# Re-derives the layer-1/layer-2 fixtures from the pinned upstream Maestro
|
|
# artifacts and fails if the checked-in files differ by a single byte.
|
|
#
|
|
# Why this exists as its own job: per-PR CI must stay Java-free (ADR 0015), so it
|
|
# can only recompute the fixtures' content seal. That seal is tamper-EVIDENT — it
|
|
# stops accidental or casual hand-editing — but someone could edit a capture and
|
|
# recompute the hash. This job closes that hole for real: forgery cannot survive
|
|
# an actual re-derivation against the pinned jars. It is what makes "generated
|
|
# from upstream" an enforced property rather than a claim in a README.
|
|
#
|
|
# Scheduled + manual only. It needs a JDK, Gradle, and network access to Maven
|
|
# Central, none of which belong on the per-PR path.
|
|
|
|
on:
|
|
schedule:
|
|
- cron: '0 6 * * *'
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ci-${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
regenerate-verify:
|
|
name: Regenerate & Verify Fixtures
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
# regenerate.mjs itself needs only Node builtins, but the verify step below
|
|
# parses corpus flows with the live engine, which imports the `yaml`
|
|
# package — so deps must be installed here.
|
|
- name: Setup toolchain
|
|
uses: ./.github/actions/setup-node-pnpm
|
|
|
|
- name: Setup JDK
|
|
uses: actions/setup-java@c1e323688fd81a25caa38c78aa6df2d33d3e20d9 # v4
|
|
with:
|
|
distribution: temurin
|
|
java-version: '17'
|
|
|
|
# Verifies the pinned jar SHA-256s against pinned-upstream.json, runs the
|
|
# harness over the corpus, and rewrites fixtures/ + corpus/manifest.json.
|
|
- name: Regenerate fixtures from pinned upstream
|
|
run: pnpm maestro:conformance:regenerate
|
|
|
|
- name: Fail if regeneration changed anything
|
|
run: |
|
|
if ! git diff --exit-code -- scripts/maestro-conformance/fixtures scripts/maestro-conformance/corpus/manifest.json; then
|
|
echo "::error::Checked-in conformance fixtures do not match a fresh regeneration from the pinned upstream artifacts."
|
|
echo "The fixtures are generated: run 'pnpm maestro:conformance:regenerate' and commit the result."
|
|
exit 1
|
|
fi
|
|
echo "Fixtures are byte-identical to a fresh regeneration from Maestro ${{ github.sha }}."
|
|
|
|
# The seal check also runs per-PR, but assert it here too so a regeneration
|
|
# that produced an unsealed/mis-sealed fixture cannot pass silently.
|
|
- name: Verify fixture seals and conformance
|
|
run: node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test packages/maestro/test/conformance/verify.test.ts
|