mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
Michał Pierzchała
2557670193
* test: slow-test ratchet, budget-derived emulator poll, speed guidance from experiments Measured (2026-07-04, full unit suite: 340 files / 3,210 tests / 48s wall): wall clock was bounded by the slowest FILE (44.6s android monolith at ~7x file-level parallelism), and the slowest tests were sleeping through real production budgets (10.8s proving 'times out' by waiting the constant out, 8s emulator polls at 1Hz, real retry backoff). Two config experiments rejected with data: --no-isolate exploded the suite to 205s (module state thrashes across files sharing workers) and --pool=threads changed nothing. - scripts/vitest-slow-test-reporter.ts: the slow-test ratchet. Unit budget 2.5s / integration 15s; failure at 2x budget (the band between reports without failing so host-load variance cannot make the gate cry wolf); 36 pinned offenders, exact keys, ratchet-only pin (tracking #1098). - waitForAndroidEmulatorByAvdName: poll cadence derives from the caller's budget (min 1s, floor 50ms, ~timeout/20) — devices.test.ts 25.6s -> 2.8s (9x) in isolation, and short-budget production calls stop sampling at 1Hz against small budgets. - vitest.config: slowTestThreshold 500 for local visibility; reporter wired; isolation/pool decisions documented with the measurements. - docs/agents/testing.md 'Speed rules' + AGENTS.md testing bullet: the three conversion patterns in preference order (budget-derived cadence, budget-wiring assertion, fake clocks), the no-seam constraint, and the file-granularity Amdahl argument that makes the monolith test split a wall-clock fix, not just navigation. * fix: fallow findings on the slow-test gate — import edge, factory reporter, unit tests The string-path reporter wiring read as a dead file (fallow cannot see vitest's reporter loading); the config now imports the factory, making the edge real and type-checked. The class shape tripped the unused-class-members rule (framework callbacks are invisible to reference analysis) — converted to a factory returning the Reporter object, with the classification and rendering logic extracted as pure exported functions. Those functions now carry their own unit tests (budget bands, integration budgets, pin matching, warn-vs-fail rendering), which also grounds the CRAP estimate in real references. Canary re-verified: unpinned 5.2s sleeper fails the run with exit 1; clean runs exit 0.
41 lines
2.7 KiB
Markdown
41 lines
2.7 KiB
Markdown
# Testing Notes
|
||
|
||
## Live web smoke
|
||
|
||
The live web platform smoke runs the public built CLI against a local fixture page through the managed web backend:
|
||
|
||
```bash
|
||
AGENT_DEVICE_WEB_E2E=1 pnpm test:smoke:web
|
||
```
|
||
|
||
The test is skipped unless `AGENT_DEVICE_WEB_E2E=1` is set. The test runs `agent-device web setup` and `agent-device web doctor` with an isolated state directory before opening the fixture URL, so it verifies the public managed-backend setup path instead of relying on a global `agent-browser`. CI runs the lane on Node 24 because the managed backend requires Node >= 24. Failure artifacts, daemon state, and browser config are written under `test/artifacts/web/`.
|
||
|
||
## Speed rules (experiment-backed, 2026-07-04)
|
||
|
||
Measured on the full unit suite (340 files, 3,210 tests, 48s wall at ~7x parallelism):
|
||
|
||
- **Wall clock equals the slowest file.** The 44.6s android monolith bounded the whole 48s run
|
||
(Amdahl at file granularity: vitest parallelizes per file). Splitting monolith test files is a
|
||
wall-clock optimization, not just a navigation one — see the AGENTS.md test-topology mirror rule.
|
||
- **Unit tests must not wait real time.** The suite's worst tests slept through production budgets:
|
||
10.8s to prove "times out" by waiting out the full constant, 8s emulator-boot polls at 1Hz, real
|
||
retry backoffs. Conversion patterns, in preference order (tracking issue #1098):
|
||
1. *Budget-derived cadence* (production-legit): poll intervals scale with the caller's timeout —
|
||
this took `devices.test.ts` from 25.6s to 2.8s (9x) while making short-budget production calls
|
||
more responsive.
|
||
2. *Budget-wiring assertion*: don't re-prove the exec layer's timeout per call site; mock the tool
|
||
layer and assert the right `timeoutMs` constant is passed. Exec-layer timeout semantics are
|
||
proven once, in exec's own tests.
|
||
3. *Fake clocks* where the code accepts an injected clock.
|
||
Never add a test-only DI seam for this — the CI gate forbids it; patterns 1–2 are production
|
||
improvements and test restructurings respectively.
|
||
- **The slow-test ratchet** (`scripts/vitest-slow-test-reporter.ts`) enforces this: unit budget
|
||
2.5s, integration 15s, failure at 2x budget (the band between reports without failing — host
|
||
load legitimately stretches borderline tests, and a flaky gate trains people to ignore it).
|
||
The pin list only shrinks, or grows in the same PR with a justification.
|
||
- **Isolation stays ON; pool stays forks — both measured.** `--no-isolate`: 205s wall vs 48s
|
||
(module state — timers, memos, singletons — thrashes across files sharing a worker).
|
||
`--pool=threads`: no change (50.4s). The ~100s aggregate import overhead is the price of
|
||
isolation and is paid in parallel; reduce it per file by importing the module under test, not
|
||
platform barrels.
|