callstack__agent-device/docs/agents/testing.md

Testing Notes

Repository-specific testing traps you cannot learn from the test runner alone. Executable gate ownership lives in scripts/check-affected/ and scripts/gate/.

Which gates a change needs

Three tiers:

1. While editing: a focused test or pnpm check:quick.
2. Before pushing: pnpm check:affected --run. It derives the relevant local gates and lists the checks that CI or a native toolchain owns.
3. For a broad refactor, or when the full deterministic gate is requested: pnpm check.

GitHub stays authoritative for provider integration, full coverage, native builds, device lanes, and history-backed compatibility. To inspect the gate catalog or a plan:

pnpm check:affected
pnpm check:affected --json
pnpm gate --help

check:affected --run reports coverage obligations but never turns coverage instrumentation on. It runs one capped vitest related command. Run the dedicated coverage scripts only to diagnose a red CI result.

Two selection traps recur:

• A response that emits platform or appleOs needs provider integration and coverage evidence. Unit tests do not run the provider project, which is what catches internal apple leaking onto the wire.
• A workspace package manifest or TypeScript config can rewire all consumers, so the affected selector fails open to the full gate set on purpose.

Docs-only changes with no behavior impact need no runtime tests. Structural guidance gates still need a planted violation that shows their failure direction.

Platform and live-device policy

HarmonyOS has no provisioned CI emulator, physical device, DevEco image, or HDC installation. Unit, provider, and coverage tests mock the typed HDC seam. Real validation is local hardware evidence per docs/agents/device-verification.md. Do not add a CI lane that assumes a developer host.

Apple runner changes run pnpm check:xctest-selection and build the affected target. The source #if guard is the XCTest lane classification — never maintain a second test-name list. Pure runner decisions use the macOS host lane; iOS/XCTest semantics need a simulator lane.

Local host-lane XCTest runs hit two snags CI never does:

• System policy may refuse the unsigned bundle (library load disallowed by system policy, shown as Early unexpected exit … crashed with signal kill). Rebuild signed: CODE_SIGN_IDENTITY="Apple Development", or pick an identity from security find-identity -v -p codesigning.
• The first run needs XCUITest automation permission for the host app.

Live smoke commands and their environment contracts live with their harnesses:

• web: test/integration/smoke-web-platform.test.ts
• iOS: test/integration/smoke-ios-simulator.test.ts and test/integration/smoke-ios-simulator-coverage.test.ts
• concurrency: test/integration/nightly/concurrency-torture.test.ts

Read the entry file before running a lane. Do not copy its environment matrix here — it changes.

Shared test utilities

Before creating fixtures, look in src/__tests__/test-utils/. Import named builders from the module that defines them (session-factories.ts, device-fixtures.ts, store-factory.ts). There is no barrel on purpose: one barrel made every test evaluate every helper's transitive graph. Shared DeviceInfo, session, snapshot, store, runtime-fact, and mocked-binary values belong in a sibling fixture module, not in repeated test literals.

Use mkdtempForTest or mkdtempForTestSync. Global setup redirects TMPDIR for the whole run and removes it after every worker exits — do not add per-test cleanup for those directories. An interrupted run may leave a directory behind; the next run prunes it once the owner process and every process using its TMPDIR are gone.

Mock the seam the subject consumes. A daemon handler that binds a runtime gets fake runtime facts and facets, not a mock of generic dispatch. Generic dispatch mocks are migration debt — do not add more. When an ADR 0019 command migrates, its tests move to the runtime seam in the same PR.

Vitest workers may signal only themselves and their direct children; hermetic-signal-setup.ts rejects other process-table writes. Tests with fabricated PIDs mock signalPidsBestEffort, signalProcessGroupBestEffort, or the tool-provider seam. A real child the test spawned may be signalled directly.

Regression evidence

A regression test must be seen failing without the production change: revert the implementation, run the smallest owning test, record the failing count, restore. Apply the same proof to test relocation and structural gates — plant a type error or violation and watch the intended gate find and name it.

A callback-based canary must observe the subject's semantic success, not just lifecycle completion. Example: React Native Gesture Handler's onFinalize also fires when recognition fails or is interrupted — use an activation-dependent callback, or assert the callback's success state before publishing a pass.

A device replay counts as automatic regression coverage only when an automatic PR or scheduled lane selects and runs it. Name the owning lane and confirm the scenario ran on the exact PR head. A replay in a manual or unselected tier is test material, not automatic evidence.

For structured classifiers, pair the positive case with the closest negative. When an error message can be identical with and without a typed reason, the negative test must prove the message alone cannot activate retry, fallback, or recovery.

Test through public interfaces where practical. Never add production exports or test-only dependency injection just for a test; a missing seam must be a real product seam.

Properties, fuzzing, and mutation

Pure parser and geometry changes extend the shared fast-check arbitraries and properties — not just another example. Keep examples for a real past bug or a named decision. Reuse PROPERTY_RUNS budgets so property files stay inside the unit slow-test gate.

Parser fuzz targets live in scripts/fuzz/targets.ts. Validation generators carry the invalid outcome they planted, so silent acceptance and wrong error codes are failures. Cases run in a worker process, so the two faults a case cannot report about itself — never returning, and killing the process it runs in — are reported as hang and crash against the exact input rather than taking the caller down with them. Promote a discovered case with the command the harness prints — never hand-copy an unshrunk input.

Mutation is report-only and limited to the registry in scripts/mutation/modules.ts. It measures whether tests distinguish changed decision logic. Do not infer redundancy from line coverage alone.

Before editing a shared module

Run pnpm depgraph affected before touching a high-fan-in module:

pnpm depgraph affected packages/capture-kit/src/exec.ts
pnpm depgraph affected src/daemon/ref-frame.ts --json --limit 25

It reports value-edge dependents, affected gates, public commands whose handler chains reach the module, live scenario owners, and interaction-guarantee cells; type-only and dynamic edges are classified separately. Feed the plan into pnpm check:affected --run — do not keep a parallel gate list in prose.

Gate ownership

CHECK_CATALOG is the executable check registry. CI owns a check only through the shared run-gate action with a literal gate id; pnpm check:gate-manifest verifies registration, workflow ownership, path reachability, and routed device lanes. Raw shell text cannot declare ownership.

New check: update the catalog and its executable model. Changed path ownership: plant a path that would previously be misrouted and watch the selector or manifest fail before fixing it. Workflow limitations (manual-only, opaque owners) belong in the gate declarations, not here.

Concurrency torture lane

The harness uses a deterministic scheduler for modeled lock grants plus a separate real request-scope serialization guard. A seed reproduces the scheduler trace and terminal invariant:

pnpm test:concurrency-torture
TORTURE_SEED=1234 pnpm test:concurrency-torture

Lock plans come from the production request-lock decisions — never hand-author a parallel plan. The modeled boundary is documented in the harness module, and every failure prints its exact replay command.

Real-subprocess-spawn tests

SUBPROCESS_STUB_TESTS enumerates the few files that spawn a real subprocess per case. They ran serialized in their own Vitest project until #1823's kill criterion: now un-serialized in unit-core's default forks pool, reverted if a timeout-shaped failure appears within 20 consecutive CI runs. Still excluded from the mutation lane either way. There is no unit-test retry layer — fix or remove flakes.

Speed rules

• Unit tests do not wait production time. Prefer budget-derived cadence, assert the caller passes the right timeout to its tool seam, or use an existing clock seam.
• Vitest parallelizes files, so wall clock is bounded by the slowest file. Splitting a monolith along source topology is a performance win, not just a readability win.
• The slow-test reporter enforces unit and integration budgets. Existing pins only shrink; a new pin needs measured justification.
• Test files over 1,000 lines are pinned to their merge-base size and may only shrink. Split the family before adding tests; never raise the pin.
• Keep isolation enabled and the pool on forks — both alternatives were measured and did not help. The useful optimization is importing the module under test, not a platform barrel.
• Local Vitest runs use a four-worker cap. Override it when a run needs a different host share: AGENT_DEVICE_VITEST_MAX_WORKERS=<n> (clamped to host CPUs, ignored in CI).