callstack__agent-device/docs/agents/testing.md

# Testing Notes

Gate ownership lives in `scripts/check-affected/` and `scripts/gate/`.

## Which gates a change needs

Three tiers:

1. While editing: a focused test or `pnpm check:quick`.
2. On the exact pushed head (locally, or by a gate stage recording it on the PR):
   `pnpm check:affected --run`. It derives the relevant local gates and lists checks CI or a
   native toolchain owns.
3. For a broad refactor, or when the full deterministic gate is requested: `pnpm check`.

GitHub stays authoritative for provider integration, full coverage, native builds, device lanes, and
history-backed compatibility. To inspect the gate catalog or a plan:

```sh
pnpm check:affected
pnpm check:affected --json
pnpm gate --help
```

`check:affected --run` reports coverage obligations without instrumenting; when related tests are
selected, it runs one capped `vitest related` command. Run local coverage to investigate a specific
coverage question or CI failure.

Two selection traps recur:

- A response emitting `platform` or `appleOs` needs provider integration and coverage evidence;
  only the provider project catches internal `apple` leaking onto the wire.
- A workspace package manifest or TypeScript config can rewire all consumers, so the affected
  selector fails open to the full gate set on purpose.

Docs-only changes with no runtime behavior impact need no runtime tests or new tests asserting prose.
Keep required gates; after those and focused checks pass, repeat or broaden only for changes,
failures, or unresolved risks.

## Platform and live-device policy

HarmonyOS has no provisioned CI emulator, physical device, DevEco image, or HDC installation. Unit,
provider, and coverage tests mock the typed HDC seam. Real validation is local hardware evidence per
`docs/agents/device-verification.md`. Do not add a CI lane that assumes a developer host.

Apple runner changes run `pnpm check:xctest-selection` and build the affected target. The source
`#if` guard is the XCTest lane classification — never maintain a second test-name list. Pure runner
decisions use the macOS host lane; iOS/XCTest semantics need a simulator lane.

Local host-lane XCTest may need signing and automation permission:

- System policy may refuse the unsigned bundle (`library load disallowed by system policy`, shown
  as `Early unexpected exit … crashed with signal kill`). Rebuild signed:
  `CODE_SIGN_IDENTITY="Apple Development"`, or pick an identity from
  `security find-identity -v -p codesigning`.
- The first run needs XCUITest automation permission for the host app.

Live smoke commands and their environment contracts live with their harnesses:

- web: `test/integration/smoke-web-platform.test.ts`
- iOS: `test/integration/smoke-ios-simulator.test.ts` and
  `test/integration/smoke-ios-simulator-coverage.test.ts`
- concurrency: `test/integration/nightly/concurrency-torture.test.ts`

Read the entry file before running a lane. Do not copy its environment matrix here — it changes.

## Shared test utilities

Before creating fixtures, look in `src/__tests__/test-utils/`. Import named builders from the module
that defines them (`session-factories.ts`, `device-fixtures.ts`, `store-factory.ts`); avoid importing
unrelated helpers through a barrel. Shared `DeviceInfo`, session, snapshot, store, runtime-fact,
and mocked-binary values belong in a sibling fixture module, not in repeated test literals.
`PROPERTY_RUNS` and the interaction touch-point/rect arbitraries live in
`@agent-device/selectors/snapshot-geometry-fixtures` — the canonical location root tests and the
selectors package both build on — not in `src/__tests__/test-utils/`. `makeSnapshotState` itself
canonically lives in `@agent-device/capture-kit/snapshot-state-fixtures`, re-exported from there.

Use `mkdtempForTest` or `mkdtempForTestSync`. Global setup redirects `TMPDIR` for the run and
removes it after every worker exits — skip per-test cleanup. An interrupted run may leave a
directory behind; the next run prunes it once its owner and every process using its `TMPDIR` are
gone.

Mock the seam the subject consumes. A daemon handler that binds a runtime gets fake runtime facts
and facets, not a mock of generic dispatch. Generic dispatch mocks are migration debt — do not add
more. When an ADR 0019 command migrates, its tests move to the runtime seam in the same PR.

Vitest workers may signal only themselves and their direct children; `hermetic-signal-setup.ts`
rejects other process-table writes. Tests with fabricated PIDs mock `signalPidsBestEffort`,
`signalProcessGroupBestEffort`, or the tool-provider seam. A real child the test spawned may be
signalled directly.

## Regression evidence

Observe a regression test fail without the fix, then pass with it. For new or changed structural
gates, plant a violation and verify the intended gate names it. Pure test moves retain their tests;
verify discovery at the new path. If selection rules change, plant a failure to prove selection.

A callback-based canary must observe semantic success, not just lifecycle completion — e.g. React
Native Gesture Handler's
[`onFinalize`](https://docs.swmansion.com/react-native-gesture-handler/docs/fundamentals/callbacks-events/)
also fires on failed or interrupted recognition. Use an activation-dependent callback, or assert
success state before publishing a pass.

A device replay counts as automatic regression coverage only when an automatic PR or scheduled lane
selects and runs it. Name the owning lane and confirm the scenario ran on the exact PR head. A
replay in a manual or unselected tier is test material, not automatic evidence.

For structured classifiers, pair the positive case with the closest negative. When an error message
can be identical with and without a typed reason, the negative test must prove the message alone
cannot activate retry, fallback, or recovery.

Test through public interfaces where practical; never add production exports or test-only
dependency injection — a missing seam needs a real product seam.

## Properties, fuzzing, and mutation

Pure parser and geometry changes extend the shared fast-check arbitraries and properties — not just
another example. Keep examples for a real past bug or a named decision. Reuse `PROPERTY_RUNS`
budgets so property files stay inside the unit slow-test gate.

Parser fuzz targets live in `scripts/fuzz/targets.ts`. Validation generators carry the invalid
outcome they planted, so silent acceptance and wrong error codes are failures. Cases run in a
worker process, so a case that never returns or kills its process is reported as `hang` or `crash`
against the exact input. Promote a discovered case with the command the harness prints — never
hand-copy an unshrunk input.

Mutation is report-only and limited to the registry in `scripts/mutation/modules.ts`. It measures
whether tests distinguish changed decision logic. Do not infer redundancy from line coverage alone.

## Before editing a shared module

Run `pnpm depgraph affected` before touching a high-fan-in module:

```sh
pnpm depgraph affected packages/host-kit/src/command.ts
pnpm depgraph affected src/daemon/ref-frame.ts --json --limit 25
```

Use its dependent, command, and guarantee-cell report to scope inspection. `pnpm check:affected --run`
selects gates independently from the diff; it does not consume the depgraph report.

## Gate ownership

`CHECK_CATALOG` is the executable check registry. CI owns a check only through the shared `run-gate`
action with a literal gate id; `pnpm check:gate-manifest` verifies registration, workflow ownership,
path reachability, and routed device lanes. Raw shell text cannot declare ownership.

New check: update the catalog and its executable model. Changed path ownership: plant a path that
would previously be misrouted and watch the selector or manifest fail before fixing it. Workflow
limitations (manual-only, opaque owners) belong in the gate declarations, not here.

## Concurrency torture lane

The harness uses a deterministic scheduler for modeled lock grants and a real request-scope
serialization guard. A seed reproduces the scheduler trace and terminal invariant:

```sh
pnpm test:concurrency-torture
TORTURE_SEED=1234 pnpm test:concurrency-torture
```

Lock plans come from the production request-lock decisions — never hand-author a parallel plan. The
modeled boundary is documented in the harness module; every failure prints its exact replay command.

## Test evidence and versioned inputs

Run- or commit-stamped benchmark output is never committed under `scripts/`: produce it at run
time or fetch it from the evidence branch. Versioned inputs (fuzz corpus, Maestro fixtures,
schemas, `contracts/fixtures/` tables) are unaffected.

## Speed rules

Changing timeout failures that pass alone may be host contention. Reproduce on `origin/main` under
the same load before classifying them as regressions.

- Unit tests have no retry layer. Fix or remove flakes instead of hiding them behind retries.
- Unit tests do not wait production time. Prefer budget-derived cadence, assert the caller passes
  the right timeout to its tool seam, or use an existing clock seam.
- The slow-test reporter enforces unit and integration budgets. Existing pins only shrink; a new
  pin needs measured justification.
- Test files over 1,000 lines may be no longer than at the merge-base with `origin/main`, and no
  new test file may cross that line. Split the family before adding tests; shrinking needs no
  gate edit.
- Keep isolation enabled and the pool on forks; disabling isolation and changing pools did not
  improve measured performance. Import the module under test, not a platform barrel.
- Local Vitest runs use a four-worker cap. Override it when a run needs a different host share:
  `AGENT_DEVICE_VITEST_MAX_WORKERS=<n>` (clamped to host CPUs, ignored in CI).