Commit Graph

9 Commits

Author SHA1 Message Date
Michał Pierzchała 2557670193 test: slow-test ratchet and speed rules from measured experiments (#1099)
* test: slow-test ratchet, budget-derived emulator poll, speed guidance from experiments

Measured (2026-07-04, full unit suite: 340 files / 3,210 tests / 48s wall):
wall clock was bounded by the slowest FILE (44.6s android monolith at ~7x
file-level parallelism), and the slowest tests were sleeping through real
production budgets (10.8s proving 'times out' by waiting the constant out,
8s emulator polls at 1Hz, real retry backoff). Two config experiments
rejected with data: --no-isolate exploded the suite to 205s (module state
thrashes across files sharing workers) and --pool=threads changed nothing.

- scripts/vitest-slow-test-reporter.ts: the slow-test ratchet. Unit budget
  2.5s / integration 15s; failure at 2x budget (the band between reports
  without failing so host-load variance cannot make the gate cry wolf);
  36 pinned offenders, exact keys, ratchet-only pin (tracking #1098).
- waitForAndroidEmulatorByAvdName: poll cadence derives from the caller's
  budget (min 1s, floor 50ms, ~timeout/20) — devices.test.ts 25.6s -> 2.8s
  (9x) in isolation, and short-budget production calls stop sampling at
  1Hz against small budgets.
- vitest.config: slowTestThreshold 500 for local visibility; reporter
  wired; isolation/pool decisions documented with the measurements.
- docs/agents/testing.md 'Speed rules' + AGENTS.md testing bullet: the
  three conversion patterns in preference order (budget-derived cadence,
  budget-wiring assertion, fake clocks), the no-seam constraint, and the
  file-granularity Amdahl argument that makes the monolith test split a
  wall-clock fix, not just navigation.

* fix: fallow findings on the slow-test gate — import edge, factory reporter, unit tests

The string-path reporter wiring read as a dead file (fallow cannot see
vitest's reporter loading); the config now imports the factory, making
the edge real and type-checked. The class shape tripped the
unused-class-members rule (framework callbacks are invisible to
reference analysis) — converted to a factory returning the Reporter
object, with the classification and rendering logic extracted as pure
exported functions. Those functions now carry their own unit tests
(budget bands, integration budgets, pin matching, warn-vs-fail
rendering), which also grounds the CRAP estimate in real references.
Canary re-verified: unpinned 5.2s sleeper fails the run with exit 1;
clean runs exit 0.
2026-07-04 19:13:06 +02:00
Michał Pierzchała fa1b0b7c8a docs: configure agent skill conventions (#871) 2026-06-25 11:02:37 +02:00
Michał Pierzchała d8e6bb7aa7 fix: add web viewport control and screenshot aliases (#865)
* fix: add web full-page screenshots

* fix: add web viewport command and screenshot aliases

* fix: address viewport CI regressions
2026-06-24 19:35:03 +02:00
Michał Pierzchała d7beb52768 feat: expose web network dump through agent-browser (#838)
* feat: expose web network dump through agent-browser

* fix: reduce web network mapper complexity

* fix: gate web network headers by include mode

* test: assert compact web network summary

* refactor: simplify web network dump mapping

* refactor: trim web network coverage
2026-06-22 14:40:25 +02:00
Michał Pierzchała c6fd3dc972 test: add live web platform smoke (#832)
* test: add live web platform smoke

* test: harden web smoke cleanup
2026-06-21 13:05:51 +02:00
Michał Pierzchała d05cfc727b feat: manage web backend setup (#833) 2026-06-20 11:44:00 +02:00
Michał Pierzchała 4142daa0c3 docs: update internal agent guidance (#789) 2026-06-12 14:58:18 +02:00
Michał Pierzchała a0522c7bcc docs: document issue label workflow (#742) 2026-06-10 13:00:34 +02:00
Michał Pierzchała 59d28e8446 refactor: add provider-first device lab tests (#542)
* refactor: add provider-first device lab tests

* refactor: tighten device lab provider seams

* test: cover provider lab contracts

* docs: record device lab harness direction

* ci: run device lab integration tests

* test: move device lab under integration

* test: extract device lab helpers

* refactor: centralize apps filter defaults

* test: drop lab-covered unit tests

* test: fold platform happy paths into device lab

* test: reuse device lab helpers

* test: move device lab to in-process harness

* test: replace session handler cases with device lab

* test: harden device lab scenario contracts

* docs: define unit test retention policy

* test: expand provider device lab coverage

* test: harden provider device lab coverage

* test: cover manifest install and runner session contracts

* chore: remove unused provider cleanup code

* test: split android find device lab scenario

* test: track provider lab architecture progress

* test: clarify provider lab roadmap progress

* test: advance provider lab session coverage

* test: move menubar click routing to device lab

* test: move menubar snapshots to device lab

* refactor: centralize screenshot flag plumbing

* refactor: colocate screenshot flag metadata

* test: cover all public commands in device lab

* test: move macos wait success to device lab

* test: drop redundant perf and diff units

* test: move push payload paths to device lab

* test: move network parsing to device lab

* test: move log cleanup to device lab

* test: move log restart and boot to device lab

* test: move ios physical boot to device lab

* test: cover perf startup in device lab

* test: extract android and ios device lab worlds

* test: trim device lab world surface

* test: split snapshot capture unit coverage

* test: deepen device lab coverage and trim handler units

* test: clean up device lab migration scaffolding

* test: report device lab public command coverage

* refactor: make Apple provider seams semantic

* refactor: tighten device inventory and Linux provider seams

* refactor: tighten request provider scoping

* refactor: add semantic macos host provider

* test: broaden device lab find coverage

* test: cover workflow flags in device lab

* refactor: promote linux input provider seam

* test: clarify device lab flag coverage

* test: classify snapshot force-full progress

* test: enforce device lab progress in ci

* test: stabilize device lab ci

* test: move packaged metro smoke to integration

* test: drop stale provider seam coverage

* test: harden provider scope regression coverage

* refactor: remove stale platform barrels

* refactor: keep linux clipboard and screenshots semantic

* refactor: move macos host tools behind provider

* fix: honor remote artifact output paths

* test: deepen runtime coverage for daemon and runner paths

* test: share loopback test helpers

* refactor: make daemon runtime importable

* fix: honor replay target metadata

* chore: tighten final device lab quality gates

* test: share device lab setup helpers

* test: remove generic apple lab fallback

* test: deduplicate device lab helpers

* chore: tighten fallow duplication signal

* refactor: share apple diagnostic helpers

* fix: detect active android ime during fill verification

* test: consolidate provider-backed integration suite

* ci: fix fallow and iOS smoke setup

* chore: consolidate cleanup after ci fixes

* test: split vitest unit and integration projects

* docs: mention MCP discovery metadata

* docs: add agent skills context pointers

* fix: close provider recording coverage gaps

* fix: restore mcp compatibility smoke

* test: cover provider edge regressions

* test: consolidate loopback helpers

* docs: remove stale provider routing reference

* fix: harden final provider review issues

* chore: defer mcp cleanup from provider refactor
2026-05-18 14:50:52 +02:00