Files
software-mansion__argent/packages/tool-server/test/native-devtools-remote-and-flow-tree.test.ts
Jakub Piasecki 2e182f7971 fix(flows): pin iOS tree reads to the launched app instead of auto-resolving every connection (#772)
## The problem

Flow runs on CI iOS fail with `ViewInspector RPC timed out:
Application.getState` at seemingly random steps - mid-flow after typing,
on the first read after a fresh launch, in a probe flow 25 s after the
original failure - while the app under test is provably healthy:
spindumps taken at failure time show its main thread idle in the
runloop, display links firing, and the injected devtools queue empty.
The same flows pass locally.

The chain: argent's iOS injection is simulator-wide (`launchctl setenv
DYLD_INSERT_LIBRARIES` inside the simulator's launchd), so every process
the simulator spawns afterwards loads the devtools dylib, not just the
app under test. The dylib's constructor connects unconditionally (no
UIKit or `com.apple.*` guard) and the tool-server registers whatever
bundle id shakes hands, so Spotlight workers (`mdworker_shared`),
background-launched system apps doing first-boot work (`MobileCal`), and
bare daemons (nil bundle id, registered as `"unknown"`) all become
connections indistinguishable from the app under test.

Flow tree reads auto-resolved the target on every read:
`queryFullHierarchyTree` passed `undefined` to `resolveNativeTargetApp`,
which `Promise.all`s `Application.getState` over every connection. The
dylib serves `getState` with a `dispatch_sync` onto the main thread, and
a background-launched or suspended process never services its main
queue, so its `getState` parks until the 5 s RPC timeout and the
`Promise.all` rejects. One poisoned sibling connection fails every tree
read in the run; the app under test is never even asked.

That one mechanism explains everything the failures did. CI spindumps
caught the poisoner red-handed (`MobileCal` parked in
`HandleViewInspectorRequest -> CurrentApplicationState`) next to a
healthy app. "Permanent wedge" vs "recovery" was just poisoner lifetime:
a Spotlight worker exits and its connection drops with it, so reads
recover; Calendar lingers and the whole job stays poisoned, fresh app
launches and probe flows included. And it never reproduced locally
because a warm simulator did its first-boot indexing and migrations long
ago, while CI boots a pristine device and does exactly that work during
the test window - which also made the failures correlate with typing and
tapping purely by timing.

## The fix

Pin the target instead of resolving it per read. `runLaunch` records the
launched app on the run state once the tree-source gate passes, as
`ActionEnv.treeTarget` - a `FlowTreeTarget` carrying the `bundleId`,
whether the runner still vouches for it (`pinned`), and whether its
probe has ever answered in this run (`probeAnswered`). `fetchFlowTree`
threads that target down to `queryFullHierarchyTree`, the only platform
source that consumes it.

A pinned read talks to the pinned app and nobody else: it never calls
`resolveNativeTargetApp` at all, and runs four gates of its own instead
(`flow-ios-tree.ts`). A `com.apple.*` id is refused first, terminally
and before anything is spent - a system app is never the app under test.
Then `isConnected`, a map lookup, so a dead pin reports the dropped
connection instead of driving env repair from 300 ms read polls, which
could otherwise latch the device's process-wide give-up. Then a probe of
the pinned app's own `getAppState`, and only that app's: a timeout
before this pin has ever answered is ridden out as the cold start it
usually is, one after an earlier answer is refused as an app that
stopped servicing its main queue. Finally a foreground check on the
answered state, as lenient over one app as auto-resolve is, so a flow
that left the app fails with the crisp diagnosis instead of describing a
screen that is not on screen. A poisoned sibling connection is never
touched.

The pin is only trusted while the runner can vouch for what is on
screen, and it comes off in two tiers. The five
`FOREGROUND_CHANGING_TOOLS` (`launch-app`, `restart-app`,
`reinstall-app`, `open-url`, `button`) drop the target outright, because
the launched app may no longer be on screen at all; `launch-app` and
`restart-app` then re-set it from their own `bundleId` arg, unpinned,
since they name the app they switched to - which is how a recorded flow
switches apps, the recorder keeping them as raw `tool:` steps. Every
other raw `tool:` step demotes the pin to an unpinned hint, its effect
on the foreground being opaque to the runner; a launch attempt drops the
target outright until it succeeds. Either tier also retires a proven
tree-outage memo: a verdict proven against the pinned branch's four
gates says nothing about the path the next read runs.

An unpinned read is the pre-pin behavior with one arbiter added.
`resolveNativeTargetApp` fans out over every connection, and only when
that fan-out times out - the CI condition this change exists for - does
the hint take the read, and only after passing the same terminal
`com.apple.*` policy gate, the connections check, and a
foreground-checked probe of its own. A resolution that answers always
wins, including its deliberate "single app but backgrounded" refusal, so
the arbiter never overrides a guard that fired: it only rides out a
probe a stalled sibling made unanswerable.

Nested `run:` fragments share the run state, so they inherit the pin,
its demotion and its clearing. E2e flows begin with a `launch` by
definition, so every selector read in a CI run happens pinned. Only iOS
consumes the target; the other platforms' tree sources are per-device
and never auto-resolve.

The review rounds also surfaced that the `isInjectableBundleId` "library
validation, the dylib can never load" rationale was disproven by this
PR's own spindump evidence, so that text - and every other home of the
same claim, `unreadableHierarchyReason` and `treeSourceGate` included -
is rewritten as the policy it actually is (system apps are never the app
under test; their observed connections are background processes whose
reads hang or lie) with the gates' behavior unchanged.

## What this doesn't cover

Auto-resolve's own fan-out is still all-or-nothing, so every read that
reaches it without a hint the gates accept stays poisonable: top-level
fragment runs, `describe` without an explicit bundle id, and the stretch
after a raw `tool:` step whose hint is refused or cannot answer.
Recording (`captureTapSelector`) now passes the session's last `launch`
id as an unpinned hint, so it arbitrates the same timed-out fan-out, but
it never pins - the recorder cannot vouch for the foreground app the way
a `launch:` step can. Closing the rest means hardening auto-resolve
itself (exclude connections that don't answer instead of failing the
fan-out) and refusing the connections at the source (dylib constructor
bail for non-UIKit and `com.apple.*` processes, plus a server-side
handshake filter to cover already-shipped dylibs). Both are follow-ups.

## Tests

Three files, and every behavior below is mutation-proven: disabling the
seam, dropping the target from any single call site, removing an
invalidation, or removing any one of the four gates each fails at least
one dedicated test. The reviewers' repros were turned into tests rather
than argued away.

`flow-ios-tree-pinned-target.test.ts` works at the tree source. It
reproduces the failure shape - a healthy app plus a poisoned
`com.apple.mobilecal` sibling whose `getState` only times out - and
proves a pinned read probes only the pinned app while an unpinned read
still fails. It then covers the states auto-resolve could never reach:
the terminal `com.apple.*` refusal, including its position ahead of the
connections gate so a disconnected system app is not sent into a restart
loop; a dead pin, where the dropped connection is reported and no env
repair runs; both sides of the `probeAnswered` split; and the foreground
guard over the states a real device produces (`inactive` under a
permission dialog, `inactive` with no scenes, `background` still holding
a foreground-inactive scene, and `background` with none). A second block
covers the unpinned arbiter: the hint read on a timeout, refused when
its own probe answers from the background, and rethrowing the fan-out
timeout when the hint is disconnected, cannot answer, or when the
resolution failed for any other reason.

`flow-launch-pins-tree-target.test.ts` proves the runner wiring through
the real `fetchFlowTree` seam: every read after a `launch` carries the
pin across all five read paths (assert, tap, type, rotate, idle), retry
polls included; fragments inherit it; a foreground-neutral `tool:` step
demotes it and a foreground-changing one that names no app drops it
outright; a later launch re-pins on a fresh target with the ride-out
re-armed; and a launch-free run keeps the fallback.

`flow-tool-step-tree-target-e2e.test.ts` drives `createRunFlowTool` with
only `resolveService` stubbed, so the real `fetchFlowTree` and
`queryFullHierarchyTree` run. It pins what the two tiers cost against a
wedged sibling: the launched app still arbitrating across a
foreground-neutral `tool:` step, the outage memo being retired when that
step demotes the pin, the `com.apple.*` and foreground refusals staying
in force through the demote, and a `hidden:` assert failing after the
demote instead of going green against a backgrounded app.
2026-08-25 13:26:46 +02:00

216 lines
9.2 KiB
TypeScript

import { describe, it, expect, vi } from "vitest";
import type { NativeDevtoolsApi } from "../src/blueprints/native-devtools";
// Two consumers of the measured state: the full-hierarchy path flows resolve
// selectors against, and the ios-remote host's `inspectRunningApp`. Both decide
// what an agent is told to do next, and on ios-remote the running/indeterminate
// split is the ONLY distinction available — it drives `requiresRestart` and
// `describe`'s `should_restart`.
const remote = vi.hoisted(() => ({ stdout: "", calls: 0, fail: false }));
vi.mock("@argent/native-devtools-ios", () => ({
bootstrapDylibPath: () => "/fake/dylibs/libArgentInjectionBootstrap.dylib",
bootstrapDylibPathTcp: () => "/fake/dylibs/tcp/libArgentInjectionBootstrap.dylib",
bootstrapDylibPathTvos: () => "/fake/dylibs/tvos/libArgentInjectionBootstrap.dylib",
tcpInjectionDylibs: () => [],
axServiceBinaryPath: () => "/fake/ax-service",
axServiceBinaryPathTcp: () => "/fake/ax-service-tcp",
}));
vi.mock("../src/utils/sim-remote", async (importOriginal) => ({
...(await importOriginal<typeof import("../src/utils/sim-remote")>()),
simctlSpawn: vi.fn(async () => {
remote.calls += 1;
if (remote.fail) throw new Error("tunnel down");
return { stdout: remote.stdout, stderr: "" };
}),
}));
import { remoteIosHost } from "../src/utils/ios-host";
import { queryFullHierarchyTree } from "../src/tools/flows/flow-ios-tree";
import type { DeviceInfo, Registry } from "@argent/registry";
const UDID = "AAAAAAAA-1111-2222-3333-444444444444";
const BUNDLE = "com.example.app";
const DEVICE: DeviceInfo = { id: UDID, platform: "ios", kind: "simulator" };
// The launched app as the runner hands it over once a raw `tool:` step has
// spent the pin.
const UNPINNED = { bundleId: BUNDLE, pinned: false, probeAnswered: false };
describe("remoteIosHost.inspectRunningApp", () => {
it("reports running-ness from the orchestrator and leaves the process unknown", async () => {
remote.stdout = `4242\t0\tUIKitApplication:${BUNDLE}[dffa][rb-legacy]\n`;
remote.calls = 0;
const inspection = await remoteIosHost.inspectRunningApp(UDID, BUNDLE);
// `running` must come off the real row: assuming true makes every stopped
// remote app `indeterminate`, answered with a restart of an app that is not
// there, and assuming false makes every running one `not_running`.
expect(inspection.running).toBe(true);
// App processes live on the orchestrator, so the local process table has
// nothing to say — a fabricated process here would be judged against this
// listener and reported as a definite verdict.
expect(inspection.process).toBeNull();
expect(remote.calls).toBe(1);
});
it("reports not running when no row backs the bundle", async () => {
remote.stdout = `4242\t0\tUIKitApplication:com.other.app[dffa][rb-legacy]\n`;
await expect(remoteIosHost.inspectRunningApp(UDID, BUNDLE)).resolves.toEqual({
running: false,
process: null,
});
});
});
describe("queryFullHierarchyTree surfaces the measured diagnosis", () => {
// Both accessors come off ONE connected set, as the real factory derives them
// from its `connections` map. Stubbing them independently lets a test assert a
// pairing the service cannot produce — a listed-but-unconnected app — so a
// diagnosis reached only that way is reached only by tests. Every unconnected
// state means an EMPTY set, hence an app auto-targeting cannot name; the id the
// flow launched is what makes it nameable.
function registryWith(connected: string[], overrides: Partial<NativeDevtoolsApi> = {}): Registry {
const api = {
listConnectedBundleIds: () => connected,
isConnected: (bundleId: string) => connected.includes(bundleId),
getAppState: async (bundleId: string) => ({
bundleId,
applicationState: "active",
foregroundActiveSceneCount: 1,
foregroundInactiveSceneCount: 0,
backgroundSceneCount: 0,
unattachedSceneCount: 0,
isFrontmostCandidate: true,
}),
appConnectionState: async (bundleId: string) =>
connected.includes(bundleId) ? "connected" : "unregistered",
...overrides,
} as unknown as NativeDevtoolsApi;
return { resolveService: async () => api } as unknown as Registry;
}
it("raises the state's own remedy rather than a blanket relaunch", async () => {
// `unregistered` is the case that matters: telling a flow author to relaunch
// here sends them round a loop the app cannot exit.
const registry = registryWith([]);
await expect(queryFullHierarchyTree(registry, DEVICE, UNPINNED)).rejects.toThrow(
/argent server stop && argent server start --detach/
);
await expect(queryFullHierarchyTree(registry, DEVICE, UNPINNED)).rejects.not.toThrow(
/relaunch it/
);
});
it("keeps the auto-target error when the run never launched an app", async () => {
// A fragment brings the device to its entry state out of band, so there is
// no launched id to measure and the auto-target error — which names its own
// next steps — is the best answer available.
const registry = registryWith([]);
await expect(queryFullHierarchyTree(registry, DEVICE)).rejects.toThrow(
/No native-devtools-connected apps are available for auto-targeting/
);
});
it("does not measure when auto-targeting resolved an app", async () => {
// The measurement is for the app the flow launched, not a second-guess of a
// resolution that succeeded. The stub serves a real window because an empty
// `windows` is its own error and would end the call before the assertion.
const appConnectionState = vi.fn(async () => "connected" as const);
const registry = registryWith([BUNDLE], {
appConnectionState,
queryViewHierarchy: async () => ({
windows: [
{
className: "UIWindow",
frame: { x: 0, y: 0, width: 400, height: 800 },
windowFrame: { x: 0, y: 0, width: 400, height: 800 },
children: [],
},
],
}),
} as unknown as Partial<NativeDevtoolsApi>);
await queryFullHierarchyTree(registry, DEVICE, UNPINNED);
expect(appConnectionState).not.toHaveBeenCalled();
});
it("degrades a rejected measurement instead of leaking the subprocess error", async () => {
// The measurement re-applies the launchd env before it can answer, so a sim
// that goes away mid-run rejects here. The other consumers degrade to
// `indeterminate`; a raw `Command failed: xcrun simctl spawn …` carries none
// of the diagnosis's guidance.
const registry = registryWith([], {
appConnectionState: async () => {
throw new Error("Command failed: xcrun simctl spawn UDID launchctl setenv");
},
});
await expect(queryFullHierarchyTree(registry, DEVICE, UNPINNED)).rejects.toThrow(
/could not be inspected/
);
await expect(queryFullHierarchyTree(registry, DEVICE, UNPINNED)).rejects.not.toThrow(
/Command failed/
);
});
// Reachable, though it reads as dead: `appConnectionState` re-reads the live
// connections map AFTER its env re-apply and process probe — several simctl
// round-trips past the empty list that routed the call here — precisely so a
// dial landing in that window is not reported as an app the service never
// registered. Folding the case away leaves it throwing a sentence that names
// no app and no remedy.
it("says the connection arrived mid-read when the measurement finds it connected", async () => {
const registry = registryWith([], { appConnectionState: async () => "connected" });
await expect(queryFullHierarchyTree(registry, DEVICE, UNPINNED)).rejects.toThrow(
/connection arrived mid-read/
);
// It must still name the app and an action; the measured states' remedies
// do not apply, because nothing is wrong with the app.
await expect(queryFullHierarchyTree(registry, DEVICE, UNPINNED)).rejects.toThrow(
new RegExp(`${BUNDLE}[\\s\\S]*Retry`)
);
await expect(queryFullHierarchyTree(registry, DEVICE, UNPINNED)).rejects.not.toThrow(
/argent server stop|restart-app/
);
});
it("gives a system app the terminal reason instead of a measured remedy", async () => {
// The launch gate lets a refused app through so a coordinate-driven flow
// still runs; selector resolution is where that refusal bites, so this is
// where it is said - and said terminally, since every measured state's
// remedy is a retry of something.
const registry = registryWith([]);
await expect(
queryFullHierarchyTree(registry, DEVICE, {
bundleId: "com.apple.Preferences",
pinned: false,
probeAnswered: false,
})
).rejects.toThrow(/Apple system app/);
await expect(
queryFullHierarchyTree(registry, DEVICE, {
bundleId: "com.apple.Preferences",
pinned: false,
probeAnswered: false,
})
).rejects.not.toThrow(/argent server stop|restart-app/);
// The remedy has to name a step form that exists AND reads no tree.
await expect(
queryFullHierarchyTree(registry, DEVICE, {
bundleId: "com.apple.Preferences",
pinned: false,
probeAnswered: false,
})
).rejects.toThrow(/tap: \{ x: [\d.]+, y: [\d.]+ \}/);
});
});