callstack__agent-device/docs/agents/testing.md

Testing Notes

Affected-check selector (pnpm check:affected)

pnpm check:affected --base <ref> derives which local checks a diff needs, so agents stop interpreting the testing matrix by hand. It is a fail-open advisory: existing GitHub CI stays authoritative and required, and this only narrows the local feedback loop.

pnpm check:affected --base origin/main --run     # default agent loop: plan + run
pnpm check:affected --base origin/main           # human-readable plan only
pnpm check:affected --base origin/main --json    # machine-readable plan only

The selection is derived from repository sources of truth rather than a hand-maintained path map:

• Affected Vitest tests are delegated to vitest related --run, using Vitest's own project configuration and static module graph. The selector passes its complete changed-file set instead of reproducing Vitest globs or import ownership. Dynamic-import relationships remain outside Vitest's analysis; GitHub's authoritative full suites still cover that boundary.
• Non-Vitest suites retain explicit ownership. Root test/integration/*.ts files use the Node integration lane, SkillGym owns its harness and skill guidance, and platform/build tools keep their native gates.
• Always-on gates (lint, typecheck, layering, fallow, format) fire for their input categories and are never silently skipped. Platform source also selects the provider-integration and coverage gates required by the Testing Matrix.
• Commands are resolved from real package.json scripts, so a renamed script fails loudly instead of dropping a gate.
• A small explicit build-ownership layer covers the paths whose owning build cannot be derived: Swift runner, Android helpers, macOS helper, MCP metadata, and the public package surface (itself derived from package.json exports).
• SkillGym ownership covers skill guidance (skills/) and the SkillGym harness (test/skillgym/) — those changes select the (local-only) SkillGym suite, and their Markdown is treated as skill/harness input, not inert docs.

Changed-file discovery folds working-tree state into the local plan: in the default local mode (--head HEAD) it unions the committed base..HEAD diff with staged, unstaged, and untracked files, and disables rename detection so both sides of a rename are classified (a moved file cannot look docs-only by its destination alone).

Anything the selector cannot classify — unknown, ambiguous, workflow/tooling, or a change to the selector's own sources (including the AGENTS.md Testing Matrix) — fails open to the full check set. The plan documents the rule and changed path behind every selected check.

Model and catalog live under scripts/check-affected/; the derivation is guarded by pnpm check:affected:test (the Affected-check Selector CI job).

Live web smoke

The live web platform smoke runs the public built CLI against a local fixture page through the managed web backend:

AGENT_DEVICE_WEB_E2E=1 pnpm test:smoke:web

The test is skipped unless AGENT_DEVICE_WEB_E2E=1 is set. The test runs agent-device web setup and agent-device web doctor with an isolated state directory before opening the fixture URL, so it verifies the public managed-backend setup path instead of relying on a global agent-browser. CI runs the lane on Node 24 because the managed backend requires Node >= 24. Failure artifacts, daemon state, and browser config are written under test/artifacts/web/.

Speed rules (experiment-backed, 2026-07-04)

Measured on the full unit suite (340 files, 3,210 tests, 48s wall at ~7x parallelism):

• Wall clock equals the slowest file. The 44.6s android monolith bounded the whole 48s run (Amdahl at file granularity: vitest parallelizes per file). Splitting monolith test files is a wall-clock optimization, not just a navigation one — see the AGENTS.md test-topology mirror rule.
• Unit tests must not wait real time. The suite's worst tests slept through production budgets: 10.8s to prove "times out" by waiting out the full constant, 8s emulator-boot polls at 1Hz, real retry backoffs. Conversion patterns, in preference order (tracking issue #1098):
  1. Budget-derived cadence (production-legit): poll intervals scale with the caller's timeout — this took devices.test.ts from 25.6s to 2.8s (9x) while making short-budget production calls more responsive.
  2. Budget-wiring assertion: don't re-prove the exec layer's timeout per call site; mock the tool layer and assert the right timeoutMs constant is passed. Exec-layer timeout semantics are proven once, in exec's own tests.
  3. Fake clocks where the code accepts an injected clock. Never add a test-only DI seam for this — the CI gate forbids it; patterns 1–2 are production improvements and test restructurings respectively.
• The slow-test ratchet (scripts/vitest-slow-test-reporter.ts) enforces this: unit budget 2.5s, integration 15s, failure at 2x budget (the band between reports without failing — host load legitimately stretches borderline tests, and a flaky gate trains people to ignore it). The pin list only shrinks, or grows in the same PR with a justification.
• Isolation stays ON; pool stays forks — both measured. --no-isolate: 205s wall vs 48s (module state — timers, memos, singletons — thrashes across files sharing a worker). --pool=threads: no change (50.4s). The ~100s aggregate import overhead is the price of isolation and is paid in parallel; reduce it per file by importing the module under test, not platform barrels.