Commit Graph

13 Commits

Author SHA1 Message Date
Michał Pierzchała b8172e3da3 perf(daemon): offload PNG decode/encode and screenshot diff to a worker thread (#734)
* perf(daemon): offload PNG decode/encode and screenshot pixel diff to a worker thread

PNG decode (inflateSync) and per-pixel screenshot diffing previously ran
synchronously on the daemon event loop, stalling all concurrent sessions
while multi-MB screenshots were processed.

- add src/utils/png-worker.ts worker_threads entry (rslib internal/png-worker)
  handling one decode, encode, or diff-pixels job per message
- add src/utils/png-worker-client.ts async wrappers (decodePngAsync,
  encodePngAsync, computeScreenshotDiffPixelsAsync) that lazily spawn the
  worker, resolve it next to the current module in dev (.ts) and dist (.js)
  like the companion tunnel entry, and fall back to the in-process
  synchronous path when the worker is unavailable
- extract the unchanged pixel-compare loop into
  src/utils/screenshot-diff-pixels.ts so both paths share identical logic
- route daemon call sites (screenshot-overlay annotate, compareScreenshots)
  through the async wrappers; results stay byte-identical

https://claude.ai/code/session_01LXZXzxi55sZ11DSyqWyBA2

* refactor(daemon): apply review findings to PNG worker offload

- guard postMessage failures: a job-specific send error (e.g. DataCloneError)
  now cleans up its pending entry and falls back to sync for that call only
- resolve the worker entry via a shared src/utils/internal-entry.ts helper
  that returns null on any resolution failure (non-file import.meta.url)
- match repo precedent: spawn the worker with --experimental-strip-types
  when the resolved entry is a .ts module
- report permanent worker degradation once (scoped diagnostic + process
  warning with the failure reason) instead of silently going sync-only
- daemon lifecycle: pre-warm the worker at startup and terminate it during
  shutdown with a 1s best-effort timeout (daemon-only)
- collapse the three async wrappers onto one kind-typed job runner with a
  single unavailability channel (rejection, no null path) and drop the dead
  mismatched-result guards
- derive the diff-pixels contract types from screenshot-diff-pixels.ts and
  share toBuffer via the contract module
- serialize worker errors with normalizeError and reconstruct AppError
  (code/message/details) client-side; the worker reuses decodePng so decode
  failures are identical on both paths
- transfer result buffers back to the client when a view fully owns its
  ArrayBuffer; clone pooled buffers to protect Node's shared buffer pool
- decode baseline/current screenshots concurrently in compareScreenshots
- move resizePngFileToMaxSize to src/utils/png-resize.ts and route its
  decode/encode through the worker (daemon screenshot --max-size path)

https://claude.ai/code/session_01LXZXzxi55sZ11DSyqWyBA2

* fix(fallow): declare png worker entry and simplify resultTransferList

The Fallow audit flagged src/utils/png-worker.ts as unreachable from any
entry point and resultTransferList as over the complexity threshold.

- Declare src/utils/png-worker.ts in .fallowrc.json's entry list: it is a
  worker_threads entry point loaded at runtime (mirrors src/daemon.ts /
  src/companion-tunnel.ts and the internal/png-worker rslib entry).
- Flatten resultTransferList into filter/map over an extracted
  ownsEntireArrayBuffer predicate, preserving the exact transfer rule
  (byteOffset === 0, byteLength === owner.byteLength, real ArrayBuffer).
- Add direct unit coverage asserting fully-owned buffers are transferred
  while pooled/offset views are not.

https://claude.ai/code/session_01LXZXzxi55sZ11DSyqWyBA2

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-10 12:27:44 +02:00
Michał Pierzchała 45cfad5cc5 feat: e2e command perf benchmark harness + nightly CI (#630)
* feat: add e2e command perf benchmark harness + nightly CI

Adds scripts/perf, a cheap end-to-end perf benchmark that drives the built
CLI through an ordered Settings tour of ~24 commands for N rounds, on a fully
isolated daemon/state-dir and self-cleaning device, and emits JSON + Markdown
reports. Per-command timing comes from wrapping each batchable command in its
own single-step batch (daemon durationMs) plus wall-clock around the process.

Wires a scheduled + workflow_dispatch CI job (perf-nightly.yml) that reuses the
cached iOS XCUITest runner (setup-apple-replay) and the Android replay host, and
runs the CLI from source via --experimental-strip-types (no dist build).

* refactor(perf): drive the harness CLI via runCmdSync, not spawnSync

Review (P2): repo rule is to spawn processes through src/utils/exec.ts, not
node:child_process directly. Switch the perf harness's invokeCli to runCmdSync
(allowFailure so non-zero exits are recorded as samples) and add a maxBuffer
option to ExecOptions/runCmdSync (snapshot payloads exceed Node's ~1MB default).

* perf(harness): warm the runner after open so the first measured command is clean

The first interaction after open/relaunch pays the one-time iOS XCUITest runner
startup (~10s+ cold) and a per-relaunch first-AX-query settle cost (~4s). That was
landing on the first measured command each round (snapshot -i), inflating it ~10x
vs the next snapshot. Run an untimed warmup snapshot -i after establishSession, after
each round's reset-open, and after every freshRoot relaunch, so no measured command
absorbs runner startup. Noted in the report header.

* refactor(perf): address review + fix Fallow CI

- exec.ts: extract spawnRejectionError + commandCloseFailure helpers, deduping the
  error/close handler clones (Fallow duplication ✗ that surfaced once the maxBuffer
  change pulled exec.ts into the audit scope).
- .fallowrc: exclude scripts/perf/** (non-shipped benchmark tooling, like examples/
  test-app) so its naturally-moderate functions don't trip the complexity gate.
- config.ts: drop unused exports CLI_BIN/DEFAULT_OUT_DIR; add readIntValue so
  --n/--rounds/--warmup report the actual flag + reject non-integers clearly.
- harness.ts: extract toSample(); type sampleError param as CliResult.
- scenario.ts: ScenarioStep is now a discriminated union on execMode (removes step.step!/
  step.args ?? []).
- comment/legend rewords (platform defaults are local-convenience/CI-overridden;
  elements = node count). check:fallow now green; typecheck/lint/unit pass.

* perf(harness): downgrade sample ok when a batch step reports ok:false

Defensive belt-and-suspenders for the Codex review note: stop-only batch already
surfaces a failed step as a top-level failure (caught by invokeCli), but if an
on-error=continue mode ever keeps the batch ok while a step fails, don't silently
count that step as a successful sample — derive ok from the step's own result.ok.
2026-05-31 14:37:59 +02:00
Michał Pierzchała ab760b6cf9 fix: apply app icon to iOS UI test runner (#611) 2026-05-29 09:16:15 +02:00
Michał Pierzchała 094c290703 perf: speed up iOS replay runner (#557)
* perf: speed up iOS replay runner

* fix: harden ios replay fast paths

* fix: address ci validation failures

* refactor: trim unused ios replay surface
2026-05-19 11:36:48 +02:00
Michał Pierzchała 59d28e8446 refactor: add provider-first device lab tests (#542)
* refactor: add provider-first device lab tests

* refactor: tighten device lab provider seams

* test: cover provider lab contracts

* docs: record device lab harness direction

* ci: run device lab integration tests

* test: move device lab under integration

* test: extract device lab helpers

* refactor: centralize apps filter defaults

* test: drop lab-covered unit tests

* test: fold platform happy paths into device lab

* test: reuse device lab helpers

* test: move device lab to in-process harness

* test: replace session handler cases with device lab

* test: harden device lab scenario contracts

* docs: define unit test retention policy

* test: expand provider device lab coverage

* test: harden provider device lab coverage

* test: cover manifest install and runner session contracts

* chore: remove unused provider cleanup code

* test: split android find device lab scenario

* test: track provider lab architecture progress

* test: clarify provider lab roadmap progress

* test: advance provider lab session coverage

* test: move menubar click routing to device lab

* test: move menubar snapshots to device lab

* refactor: centralize screenshot flag plumbing

* refactor: colocate screenshot flag metadata

* test: cover all public commands in device lab

* test: move macos wait success to device lab

* test: drop redundant perf and diff units

* test: move push payload paths to device lab

* test: move network parsing to device lab

* test: move log cleanup to device lab

* test: move log restart and boot to device lab

* test: move ios physical boot to device lab

* test: cover perf startup in device lab

* test: extract android and ios device lab worlds

* test: trim device lab world surface

* test: split snapshot capture unit coverage

* test: deepen device lab coverage and trim handler units

* test: clean up device lab migration scaffolding

* test: report device lab public command coverage

* refactor: make Apple provider seams semantic

* refactor: tighten device inventory and Linux provider seams

* refactor: tighten request provider scoping

* refactor: add semantic macos host provider

* test: broaden device lab find coverage

* test: cover workflow flags in device lab

* refactor: promote linux input provider seam

* test: clarify device lab flag coverage

* test: classify snapshot force-full progress

* test: enforce device lab progress in ci

* test: stabilize device lab ci

* test: move packaged metro smoke to integration

* test: drop stale provider seam coverage

* test: harden provider scope regression coverage

* refactor: remove stale platform barrels

* refactor: keep linux clipboard and screenshots semantic

* refactor: move macos host tools behind provider

* fix: honor remote artifact output paths

* test: deepen runtime coverage for daemon and runner paths

* test: share loopback test helpers

* refactor: make daemon runtime importable

* fix: honor replay target metadata

* chore: tighten final device lab quality gates

* test: share device lab setup helpers

* test: remove generic apple lab fallback

* test: deduplicate device lab helpers

* chore: tighten fallow duplication signal

* refactor: share apple diagnostic helpers

* fix: detect active android ime during fill verification

* test: consolidate provider-backed integration suite

* ci: fix fallow and iOS smoke setup

* chore: consolidate cleanup after ci fixes

* test: split vitest unit and integration projects

* docs: mention MCP discovery metadata

* docs: add agent skills context pointers

* fix: close provider recording coverage gaps

* fix: restore mcp compatibility smoke

* test: cover provider edge regressions

* test: consolidate loopback helpers

* docs: remove stale provider routing reference

* fix: harden final provider review issues

* chore: defer mcp cleanup from provider refactor
2026-05-18 14:50:52 +02:00
Michał Pierzchała c1550721ce fix: collapse Android snapshot row noise (#545)
* fix: collapse Android snapshot row noise

* docs: clarify agent-facing snapshot text

* fix: preserve semantics when collapsing Android snapshots

* fix: trim redundant Android collapse hints

* fix: collapse passive Android duplicate children

* refactor: drop unused Android snapshot hint code

* docs: clarify Android Metro verification guidance

* test: cover Android Metro reverse planning

* test: fix codex skillgym runner

* fix: satisfy fallow snapshot checks

* docs: clarify PR validation guidance
2026-05-15 18:52:11 +02:00
Hwidong Bae 2e04edd869 fix: prevent runner XCTest attachment bloat (#520)
Long-running AgentDeviceRunnerUITests sessions can inherit Xcode 26 screenRecording/deleteOnSuccess defaults and leave hidden testmanagerd attachments behind when sessions terminate outside a clean XCTest success path.

Pin an explicit AgentDeviceRunnerUITests.xctestplan with screenshots and keepNever attachment lifetimes, then normalize the generated per-session .xctestrun copy before test-without-building. The .xctestrun normalization protects stale cached artifacts and Xcode versions that ignore some test-plan attachment lifetime keys during build-for-testing.

This only affects XCTest's automatic attachments; user-requested agent-device recordings still use the existing record command paths.

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
2026-05-14 14:24:33 +02:00
Michał Pierzchała 8aa4abe465 feat: expand android adb provider boundary (#481) 2026-04-29 21:45:42 -04:00
Michał Pierzchała fea7a5b73f feat: expose daemon embedding and Android ADB APIs (#480) 2026-04-29 19:18:17 -04:00
Michał Pierzchała 72ba612e83 fix: centralize Android adb execution (#478) 2026-04-29 18:25:21 -04:00
Michał Pierzchała d77a9211ab feat: add Android snapshot helper (#454)
* feat: add android snapshot helper

* fix: harden android snapshot helper packaging

* fix: harden android xml attribute parsing

* fix: capture android helper window roots

* fix: validate android helper install args

* fix: harden android helper manifest and traversal

* refactor: drop unused helper timeout parameter

* feat: bundle android snapshot helper in npm package

* feat: enable bundled android snapshot helper by default

* refactor: simplify android snapshot helper resolution

* fix: harden android snapshot helper artifacts

* refactor: split android snapshot helper modules

* fix: use type-only snapshot helper imports
2026-04-27 16:01:37 -04:00
Michał Pierzchała 7c5b7670c8 feat: add skillgym tests (#453) 2026-04-26 20:49:59 -04:00
Michał Pierzchała f07e82e308 chore: add fallow quality gate (#449)
* chore: add fallow quality gate

* fix: refresh pnpm lockfile
2026-04-26 13:53:35 -04:00